A point cloud segmentation method, device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a point cloud segmentation method and device, electronic equipment and storage medium. The method is used to improve the accuracy of point cloud segmentation. The method comprises: obtaining point cloud data of a target region; preprocessing the point cloud data to obtain a target point cloud; wherein the number of points in the target point cloud is less than the number of points in the target region; calculating the normal vector and residual error of each point after preprocessing; wherein the normal vector is a vector represented by a straight line perpendicular to the plane where the point is located, and the residual error is used to indicate the curvature of the surface where each point is located; and performing plane segmentation on the target region according to the normal vector and residual error of each point.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud data segmentation, and in particular to a point cloud segmentation method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of Internet e-commerce, the entire logistics industry is facing great challenges. In this environment of large quantities, strong repetition and poor working environment, the cost of machinery is much lower than that of manual labor. Therefore, implementing robot sorting in logistics is the trend of current technology development.

[0003] At present, most robots sort by obtaining data through a camera and performing segmentation through image processing. The most commonly used image processing method is to obtain point cloud information through a depth sensor and directly perform point cloud segmentation on the obtained point cloud information. However, the accuracy of the depth sensor and the noise contained in the point cloud data will greatly affect the segmentation effect. For example, when abnormal values occur due to the low accuracy of the depth sensor, or when there are a large number of outliers in the point cloud data, the accuracy of the segmentation will be low.

[0004] Therefore, how to improve the accuracy of point cloud segmentation has become a technical problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a point cloud segmentation method and device, an electronic device and a storage medium. The accuracy of point cloud segmentation is improved.

[0006] In a first aspect, a point cloud segmentation method is provided. The method comprises:

[0007] obtaining point cloud data of a target region;

[0008] preprocessing the point cloud data to obtain target point cloud; wherein the number of points in the target point cloud is less than the number of points in the target region;

[0009] calculating the normal vector and the residual error of each point after preprocessing; wherein the normal vector is a vector represented by a straight line perpendicular to the plane where the point is located, and the residual error is used to indicate the curvature of the surface where each point is located;

[0010] performing plane segmentation on the target region according to the normal vector and the residual error of each point.

[0011] Optionally, the preprocessing of the point cloud data comprises:

[0012] performing down-sampling processing on the point cloud data to obtain first point cloud data;

[0013] filtering out the outlier noise in the first point cloud data to obtain second point cloud data;

[0014] performing mean filtering and median filtering on the second point cloud data to obtain third point cloud data;

[0015] performing an erosion operation on the third point cloud data to obtain preprocessed point cloud data.

[0016] Optionally, the outlier noise in the first point cloud data is filtered out, comprising:

[0017] calculating the density of the first point cloud data;

[0018] filtering out point cloud with density lower than a first preset threshold.

[0019] Optionally, the mean filtering and median filtering are performed on the second point cloud data to obtain third point cloud data, comprising:

[0020] performing mean filtering on the data in the second point cloud data to obtain a first coordinate value of each point, and performing median filtering on the data to obtain a second coordinate value of each point;

[0021] comparing the first coordinate value and the second coordinate value of the same point;

[0022] determining the smaller coordinate value as the coordinate value of the corresponding point after filtering.

[0023] Optionally, the target region is segmented according to the normal vector and the residual of each point, comprising:

[0024] determining a first seed point Q1 with the smallest residual from the target point cloud;

[0025] determining a point satisfying a preset condition from the target point cloud with the first seed point Q1 as the center, the point Q1 and the point satisfying the preset condition forming a first plane, the first plane being a first plane segmented from the target region;

[0026] determining a second seed point Q2 in the remaining point cloud; wherein the remaining point cloud is a point cloud excluding the points contained in the first plane from the target point cloud;

[0027] stopping the plane segmentation when the number of remaining points in the target point cloud is less than a second preset threshold.

[0028] Optionally, the determining of the point satisfying the preset condition from the target point cloud comprises:

[0029] calculating the included angle between the normal vector of the other point in the first region excluding the point Q1 and the normal vector of the point Q1; wherein the first region is a circular region with the point Q1 as the center and a preset size as the radius;

[0030] The point with an included angle less than the preset angle threshold is determined as the point satisfying the preset condition.

[0031] Optionally, the method further comprises:

[0032] The concave-convex property of each point N in the points with an included angle less than the preset angle threshold is determined, the concave-convex property being used to indicate whether a face where the point N is located is a convex face or a concave face.

[0033] The point with a concave-convex property of convexity is determined as a point on the same plane as the point Q1, to obtain a first plane.

[0034] Optionally, the concave-convex property of each point N in the point cloud with an included angle less than the preset angle threshold is determined, comprising:

[0035] The concave-convex property is determined by the following formula:

[0036] n1v-n2v=α

[0037] Wherein, n1 and n2 are normal vectors of the point Q1 and the point N respectively, and v is a directional vector of the point N to the point Q1.

[0038] When the α is greater than a third preset threshold, the point N is determined as a convex point, and when the α is less than the third preset threshold, the point N is determined as a concave point.

[0039] Optionally, the normal vector and the residual error of each point after preprocessing are calculated, comprising:

[0040] A fitting plane at the point M after preprocessing is determined by a least square method.

[0041] According to the fitting plane, a normal vector and a residual error at the point M are determined.

[0042] In a second aspect, a point cloud segmentation device is provided, the device comprising:

[0043] An acquisition module is configured to acquire point cloud data of a target region.

[0044] A processing module is configured to preprocess the point cloud data to obtain a target point cloud, wherein a number of points in the target point cloud is less than a number of points in the target region.

[0045] The processing module is further configured to calculate a normal vector and a residual error of each point after preprocessing, wherein the normal vector is a vector represented by a straight line perpendicular to a plane where the point is located, and the residual error is used to indicate a curvature of a surface where the point is located.

[0046] The processing module is further configured to perform plane segmentation on the target region according to the normal vector and the residual error of each point.

[0047] Optionally, the processing module is specifically configured to:

[0048] down-sample the point cloud data to obtain first point cloud data;

[0049] filter out outliers in the first point cloud data to obtain second point cloud data;

[0050] perform mean filtering and median filtering on the second point cloud data to obtain third point cloud data;

[0051] perform an erosion operation on the third point cloud data to obtain preprocessed point cloud data.

[0052] Optionally, the processing module is specifically configured to:

[0053] calculate the density of the first point cloud data;

[0054] filter out point clouds with a density lower than a first preset threshold.

[0055] Optionally, the processing module is specifically configured to:

[0056] perform mean filtering on data in the second point cloud data to obtain a first coordinate value of each point, and perform median filtering on the data to obtain a second coordinate value of each point;

[0057] compare the first coordinate value and the second coordinate value of the same point;

[0058] determine the smaller coordinate value as the coordinate value of the corresponding point after filtering.

[0059] Optionally, the processing module is specifically configured to:

[0060] determine a first seed point Q1 with a minimum residual from the target point cloud;

[0061] determine a point satisfying a preset condition from the target point cloud with the first seed point Q1 as the center, the point Q1 and the point satisfying the preset condition form a first plane, and the first plane is a first plane segmented from the target region;

[0062] determine a second seed point Q2 from the remaining point cloud, wherein the remaining point cloud is a point cloud excluding points included in the first plane from the target point cloud;

[0063] stop the plane segmentation when the number of remaining points in the target point cloud is less than a second preset threshold.

[0064] Optionally, the processing module is specifically configured to:

[0065] calculate an angle between a normal vector of a point other than the point Q1 in a first region and the normal vector of the point Q1, wherein the first region is a circular region with the point Q1 as a center and a preset size as a radius;

[0066] determine the point with the angle less than the preset angle threshold as the point satisfying the preset condition.

[0067] Optionally, the processing module is further configured to:

[0068] perform a concave-convex judgment on each of the points with the angle less than the preset angle threshold, the concave-convex judgment being used to indicate whether a face where the point is located is a convex face or a concave face;

[0069] determine the point with the concave-convex judgment result as a convex face as a point on the same plane as the point Q1, to obtain a first plane.

[0070] Optionally, the processing module is specifically configured to:

[0071] perform the concave-convex judgment by using the following formula:

[0072] n1v-n2v=α

[0073] wherein n1 and n2 are normal vectors of the point Q1 and the point N respectively, and v is a direction vector of the point N to the point Q1.

[0074] when the α is greater than a third preset threshold, determine the point N as a convex point, and when the α is less than the third preset threshold, determine the point N as a concave point.

[0075] Optionally, the processing module is specifically configured to:

[0076] determine a fitting plane at the preprocessed point M by using a least square method;

[0077] determine a normal vector and a residual error at the point M according to the fitting plane.

[0078] In a third aspect, an electronic device is provided, and the electronic device includes:

[0079] a memory configured to store program instructions;

[0080] a processor configured to invoke the program instructions stored in the memory, and perform steps included in the method according to the obtained program instructions.

[0081] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer executable instructions, the computer executable instructions being used to make a computer perform steps included in the method according to any one of the first aspect or the second aspect.

[0082] In a fifth aspect, a computer program product including instructions, which when executed on a computer, cause the computer to perform the point cloud segmentation method described in various possible implementation manners above.

[0083] In the embodiments of the present application, the point cloud data of the target region is obtained, and the obtained point cloud data is preprocessed to obtain a target point cloud. Finally, the target point cloud is segmented by the normal vector and the residual error of each point in the target point cloud, and the number of points in the target point cloud is less than the number of points in the target region. That is, the obtained point cloud data needs to be preprocessed before being segmented into a plane, so as to reduce the number of points in the final plane segmentation, thereby reducing the noise in the point cloud data and reducing the abnormality caused by the accuracy of the sensor, and improving the accuracy of the point cloud segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application.

[0085] Figure 1 A flowchart of a point cloud segmentation method provided by the embodiments of the present application;

[0086] Figure 2a A target scene schematic diagram provided by the embodiments of the present application;

[0087] Figure 2b A point cloud diagram of a target scene provided by the embodiments of the present application;

[0088] Figure 3 A schematic diagram of removing the background of a point cloud diagram of a target scene provided by the embodiments of the present application;

[0089] Figure 4 A down-sampling schematic diagram provided by the embodiments of the present application;

[0090] Figure 5 A before-and-after comparison schematic diagram of adaptive filtering processing provided by the embodiments of the present application;

[0091] Figure 6 A before-and-after comparison schematic diagram of an erosion operation provided by the embodiments of the present application;

[0092] Figure 7 A structural block diagram of a point cloud segmentation device provided by the embodiments of the present application;

[0093] Figure 8 A structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0094] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application. The embodiments in the present application and the features in the embodiments can be combined with each other in a non-conflicting manner. Moreover, although a logical sequence is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from the order shown.

[0095] The terms "first" and "second" in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific sequence. In addition, the term "comprising" and any variations thereof are intended to cover the non-exclusive protection. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device. "Multiple" in the present application can mean at least two, for example, can be two, three or more, and the embodiments of the present application are not limited.

[0096] In addition, the term "and / or" in the present application is only used to describe the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects without special description.

[0097] In order to facilitate understanding, the technical background of the embodiments of the present application will be introduced first.

[0098] As described above, when the point cloud is segmented, the noise and other factors in the point cloud data will affect the effect of the point cloud segmentation. At present, in order to solve the technical problem, one solution is to obtain point cloud information, remove the point cloud data of the plane where the object is located by using the Ransanc plane fitting method on the obtained point cloud information, then use LCCP to preliminarily segment the object, and perform MLS upsampling and point cloud uniformization on each segmented point cloud, and finally use CPC method to finely segment the point cloud. As can be seen, in the process of point cloud segmentation, LCCP, MLS and CPC algorithms are used, and the combination of multiple algorithms will reduce the rapidity of segmentation. Moreover, the process is too much, and the result will be affected by multiple variables, so the main parameter of the segmentation result is not obvious, and therefore it is not conducive to the user to adjust the method according to the specific scene demand.

[0099] Another solution is to obtain the normal vector and residual error of the target point included in the target region, and perform plane segmentation in the target region by using the normal vector and residual error of the target point to obtain the target plane. That is, in this solution, the normal vector and residual error of the target point are directly calculated, and then the target region is segmented by using the normal vector and residual error. However, due to the influence of the accuracy of the depth sensor and the noise in the point cloud data, the final segmentation effect will be poor, and the accuracy of the segmentation will be low.

[0100] In view of this, the present application provides a solution of a point cloud segmentation method, which needs to preprocess the point cloud data of the target region before segmenting the point cloud of the target region, so as to reduce the number of points in the plane segmentation, thereby reducing the noise in the point cloud data and reducing the abnormality caused by the accuracy of the sensor, and improving the accuracy of the point cloud segmentation. Moreover, the segmentation process is simple, and the rapidity of the segmentation can be effectively improved.

[0101] The technical solutions provided by the embodiments of the present application will be described below with reference to the accompanying drawings.

[0102] It should be noted that the point cloud segmentation method provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. The present application takes a mobile terminal, for example, a robot, as an example for description, and the application scenario of the embodiments of the present application is a logistics scenario, in which a package is grabbed by a robot.

[0103] Please refer to Figure 1 shown in Figure 1 a flowchart of a point cloud segmentation method provided by the embodiments of the present application, Figure 1 The method flowchart is as follows:

[0104] Step 101: Obtain the point cloud data of the target region.

[0105] In the embodiment of the present application, the point cloud information of the target scene can be acquired by the RGBD camera first, as shown in Figure 2a and Figure 2b . Figure 2a is a target scene provided by the embodiment of the present application, Figure 2b is a point cloud graph of a target scene provided by the embodiment of the present application. The ground surface model is removed by the background parameter model or the point cloud background model to draw the region of interest, i.e. the target region, as shown in Figure 3 . Figure 3 is a schematic diagram after the background removal of FIG. 2. As can be seen from Figure 3 , only the point cloud information outside the package is reserved after the background removal. Then the point cloud data of the target region is acquired. The RGBD camera is a camera with a depth measurement function added to the function of the RGB ordinary camera, so that the depth information of the image can be acquired.

[0106] Step 102: pre-processing the point cloud data to obtain target point cloud.

[0107] After the point cloud data of the target region is acquired, the point cloud data of the target region can be pre-processed first to reduce the number of point clouds in the target region.

[0108] In the embodiment of the present application, the pre-processing of the point cloud data of the target region includes down-sampling processing, outlier noise filtering, adaptive filtering and erosion operation, and the implementation process of each step is introduced below.

[0109] 1. Down-sampling processing

[0110] In the process of pre-processing the point cloud data of the target region, the point cloud data of the target region can be first processed by a down-sampler for down-sampling to reduce the size of the point cloud data (i.e. to reduce the number of point clouds in the target region), and a filter is used to filter the data after the down-sampling to reduce the distortion caused by aliasing in the down-sampling process, to obtain the first point cloud data, as shown in Figure 4 . Figure 4 is a schematic diagram after the down-sampling of Figure 3 . As can be seen from Figure 4 , after the down-sampling, the point cloud density is obviously reduced, and the characteristics of the package are basically reserved.

[0111] In the specific implementation process, since the number of 3D point clouds is positively correlated with the calculation time, the point cloud data of the target region is processed by the down-sampling without affecting the segmentation effect, which can effectively improve the calculation efficiency of the point cloud.

[0112] 2. Outlier noise filtering

[0113] After obtaining the first point cloud data, the point cloud density of the first point cloud data is calculated. In one possible implementation, the formula for calculating the point cloud density of the first point cloud data is as follows:

[0114]

[0115] wherein p is the density of the K-neighbor distance centered at P, N is the number of points of the K-neighbor of P, p i is the coordinate of the i-th point, and p is the coordinate of the current point.

[0116] The point cloud below the first preset threshold is regarded as outlier noise, and the outlier noise is filtered out, for example, which can also be referred to as deleting the outlier noise, to obtain second point cloud data.

[0117] 3. Adaptive filtering

[0118] After obtaining the second point cloud data, the second point cloud data is filtered by using adaptive filtering combining mean filtering and median filtering. In one possible implementation, the formula for mean filtering is as follows:

[0119]

[0120] wherein N is the number of points of the K-neighbor of P, p i is the coordinate of the i-th point, and p is the coordinate of the current point. is the coordinate of the current point P after mean filtering.

[0121] The formula for median filtering is as follows:

[0122]

[0123] wherein N is the number of points of the K-neighbor of P, p1 is the coordinate of the 1st point, p2 is the coordinate of the 2nd point, is the coordinate of the current point P after median filtering.

[0124] After the data in the second point cloud data is subjected to mean filtering, the data of each point obtained is a first coordinate value, and after the data is subjected to median filtering, the data of each point obtained is a second coordinate value. The first coordinate value and the second coordinate value of the same point, for example, P, after filtering are compared, and the smaller coordinate value is determined as the final coordinate value of P. For example, the first coordinate value of P is smaller than the second coordinate value, and then the coordinate value of P obtained after adaptive filtering is the first coordinate value. If the first coordinate value of P is greater than the second coordinate value, then the coordinate value of P obtained after adaptive filtering is the second coordinate value.

[0125] In the implementation process, the final coordinate value of each point after adaptive filtering is the value of the point with a smaller difference from the original coordinate value, that is, the coordinate value of each point in the third point cloud data obtained after adaptive filtering of the second point cloud data has a smaller difference from the original coordinate value, so that the salt and pepper noise can be avoided, and the point cloud can be smoother, please refer to Figure 5 as shown, Figure 5 The front and back comparison diagrams for adaptive filtering are shown in Figure 5 It can be seen from

[0126] 4, corrosion

[0127] After obtaining the third point cloud data, an erosion operation is performed on the third point cloud data to reduce the slight adhesion of the wrapping, please refer to Figure 6 as shown, Figure 6 The front and back comparison diagrams for the erosion operation are shown in Figure 6 It can be seen from

[0128] In the embodiment of the present application, by performing downsampling processing on the point cloud data of the target region, the number of points in the point cloud can be effectively reduced, thereby improving the point cloud computing efficiency, by filtering out the outlier noise from the first point cloud data obtained by downsampling processing, the accuracy of point cloud segmentation can be effectively improved, by performing adaptive filtering on the second point cloud data obtained after deleting the outlier noise, the salt and pepper noise can be avoided, and the point cloud can be smoother, and by performing point cloud erosion on the third point cloud data obtained by adaptive filtering, the problem of slight adhesion of the wrapping can be solved. Therefore, by preprocessing the point cloud data of the target region, the number of points in the final plane segmentation can be reduced, the noise in the point cloud data can be reduced, and the abnormality caused by the accuracy of the sensor can be reduced, thereby improving the accuracy of point cloud segmentation.

[0129] Step 103: Calculate the normal vector and residual error of each point after preprocessing.

[0130] The normal vector of a point is a vector represented by a straight line perpendicular to the plane where the point is located, and the residual error is used to indicate the curvature of the surface. The greater the residual error, the greater the change of the boundary and normal of the surface region.

[0131] In one possible implementation, the fitting plane of the target point (i.e., the point M after preprocessing) can be determined by the least square method, and the normal vector and residual error at the point M can be determined based on the fitting plane. The following describes how to determine the normal vector and residual error of the point in combination with the formula of the least square method.

[0132] The normal vector and residual error of the target point (i.e., the point M) in the K points include:

[0133] The covariance matrix ∑ of the point M is determined by the following formula:

[0134]

[0135] wherein P i is the first K / 2 points closest to the point M in the K points, is the mean point of the first K / 2 points.

[0136] The eigenvectors and eigenvalues of each point are determined by the following formula (standard eigenvalue equation):

[0137] λV = ∑V

[0138] wherein V is the eigenvector, λ is the eigenvalue, the eigenvalue with the smallest value included is determined as the residual of the point M, and the eigenvector corresponding to the smallest eigenvalue is determined as the normal vector of the point M. In the embodiment of the present application, for example, the number of obtained eigenvalues is 3, and the number of corresponding eigenvectors is also 3, if the three eigenvalues are λ0, λ1 and λ2, and λ2> λ1> λ0, then λ0 is determined as the residual of the point M, and V0 corresponding thereto is determined as the normal vector of the point M.

[0139] Step 104: performing planar segmentation on the target region according to the normal vector and the residual of each point.

[0140] In the embodiment of the present application, after the normal vector and the residual of the point M are determined, region growing is performed based on the determined normal vector and residual, that is, planar segmentation is performed on the target region, and the specific steps of planar segmentation according to the normal vector and the residual are described below.

[0141] Step S1: selecting a first seed point Q1 with the smallest residual from the target point cloud according to the calculated residual of each point.

[0142] In the embodiment of the present application, since the residual indicates the smoothness of the surface, when selecting the seed point for region growing, the point with the smallest residual is selected as the first seed point Q1, that is, region growing is started from the position with the smoothest surface.

[0143] Step S2: determining points satisfying a preset condition from the target point cloud with Q1 as the center.

[0144] The plane composed of Q1 and the points satisfying the preset condition is the first plane segmented from the target point cloud. In a possible implementation, one method of determining the points satisfying the preset condition from the target point cloud can be as follows: calculating the included angle between the normal vector of each point in a first region with Q1 as the center and a preset size as the radius and the normal vector of Q1, and the calculation formula of the included angle is as follows:

[0145] ||n Q1 ·n i || = cosθ i

[0146] Where, n Q1 It is the normal vector of point Q1, n i It is the normal vector of point i, cosθ i Cosθ is the angle between the normal vectors of point Q1 and point i. Point i whose angle is less than a preset angle threshold satisfies the preset condition. i >cosθ th The corresponding point i is a point that satisfies the preset conditions.

[0147] Step S3: Determine the concavity / convexity of points that meet the preset conditions.

[0148] In logistics scenarios, packages come in various shapes, the most common being cuboids and cubes packaged in cardboard boxes, and soft packages packaged in bags. Soft packages typically have only two sides, top and bottom, and because they are filled with material, the center is convex. Therefore, for soft packages, in one possible implementation of this application, points meeting preset conditions can be judged for convexity, and only convex points are region-grown and merged into the first plane, while concave points are considered not to belong to this plane and are not region-grown. Here, convexity indicates whether the surface where point N, one of the points meeting the preset conditions, is located is convex or concave.

[0149] The following describes a method for determining the concavity / convexity of points that meet preset conditions. The formula for determining concavity / convexity is as follows:

[0150] n1·v-n2·v=α

[0151] Where n1 is the normal vector at point Q1, n2 is the normal vector at point N, v is the direction vector from point N to point Q1, and α is the difference between the cosine of the angle between n1 and v and the cosine of the angle between n2 and v. When α > β t When α < β, point N is a convex point. t When β = 0, then point N is determined to be a concave point. Where β t The value of β can be 0, or it can be set according to precision and requirements. In this embodiment, β is not specified. t The size is limited.

[0152] Step S4: Determine the second seed point Q2 from the remaining point cloud containing the first plane in the target point cloud, and then continue to execute steps S2 and S3 to segment out the second plane from the target region.

[0153] The steps S1-S3 are executed in a loop until the number of remaining points in the target point cloud is less than a second preset threshold, for example, the number of remaining points in the target point cloud is less than 10% of the initial total number of points, and the plane segmentation is stopped.

[0154] In some other embodiments, when the point cloud segmentation of the target region is completed, the physical parameters of each segmented plane, such as length, width, height, center point, pose, etc., can also be calculated, the segmented planes are sorted, and after the sorting is completed, the reconstructed plane information can be displayed by color. The purpose of sorting the segmented planes is to determine the grasping sequence of the robot to the objects in the sorting process.

[0155] Based on the same inventive concept, the embodiments of the present application provide a point cloud segmentation device, which can realize the functions corresponding to the aforementioned point cloud segmentation method. The point cloud segmentation device can be a hardware structure, a software module, or a hardware structure plus a software module. The point cloud segmentation device can be realized by a chip system, which can be composed of a chip or can contain a chip and other discrete devices. Please refer to Figure 7 As shown in the figure, the point cloud segmentation device includes an acquisition module 701 and a processing module 702. Wherein:

[0156] The acquisition module 701 is configured to acquire point cloud data of a target region.

[0157] The processing module 702 is configured to pre-process the point cloud data to obtain a target point cloud, wherein the number of points in the target point cloud is less than the number of points in the target region.

[0158] The processing module 702 is further configured to calculate a normal vector and a residual error of each point after pre-processing, wherein the normal vector is a vector represented by a straight line perpendicular to the plane where the point is located, and the residual error is used to indicate the curvature of the surface where the point is located.

[0159] The processing module 702 is further configured to perform plane segmentation on the target region according to the normal vector and the residual error of each point.

[0160] In a possible implementation, the processing module 702 is specifically configured to:

[0161] Downsample the point cloud data to obtain first point cloud data;

[0162] Filter out outliers in the first point cloud data to obtain second point cloud data;

[0163] Perform mean filtering and median filtering on the second point cloud data to obtain third point cloud data;

[0164] perform an erosion operation on the third point cloud data to obtain preprocessed point cloud data.

[0165] In a possible implementation, the processing module 702 is specifically configured to:

[0166] calculate the density of the first point cloud data;

[0167] filter out point cloud with a density lower than a first preset threshold.

[0168] In a possible implementation, the processing module 702 is specifically configured to:

[0169] perform mean filtering on data in the second point cloud data to obtain a first coordinate value of each point, and perform median filtering on the data to obtain a second coordinate value of each point;

[0170] compare the first coordinate value and the second coordinate value of the same point;

[0171] determine the smaller coordinate value as a coordinate value of the corresponding point after filtering.

[0172] In a possible implementation, the processing module 702 is specifically configured to:

[0173] determine a first seed point Q1 with a minimum residual from the target point cloud;

[0174] determine, from the target point cloud, a point satisfying a preset condition with the first seed point Q1 as a center, the point Q1 and the point satisfying the preset condition forming a first plane, and the first plane being a first plane segmented from the target region;

[0175] determine a second seed point Q2 from a remaining point cloud; wherein the remaining point cloud is a point cloud excluding points included in the first plane from the target point cloud;

[0176] stop the plane segmentation when a number of remaining points in the target point cloud is less than a second preset threshold.

[0177] In a possible implementation, the processing module 702 is specifically configured to:

[0178] calculate an included angle between a normal vector of a point other than the point Q1 in a first region and a normal vector of the point Q1; wherein the first region is a circular region with the point Q1 as a center and a preset size as a radius;

[0179] determine the point with an included angle less than a preset angle threshold as the point satisfying the preset condition.

[0180] In a possible implementation, the processing module 702 is further configured to:

[0181] a concave-convex judgment is performed on each of the points N with the included angle less than the preset angle threshold, the concave-convex judgment being used to indicate whether a face where the point N is located is a convex face or a concave face;

[0182] the point with the judgment result of being convex is determined as a point on the same plane as the point Q1, and a first plane is obtained.

[0183] In a possible implementation, the processing module 702 is specifically configured to:

[0184] the concave-convex judgment is performed by the following formula:

[0185] n1v-n2v=α

[0186] wherein n1 and n2 are normal vectors of the point Q1 and the point N respectively, and v is a direction vector from the point N to the point Q1.

[0187] when the α is greater than a third preset threshold, the point N is determined as a convex point, and when the α is less than the third preset threshold, the point N is determined as a concave point.

[0188] In a possible implementation, the processing module 702 is specifically configured to:

[0189] the fitting plane at the preprocessed point M is determined by a least square method;

[0190] the normal vector and the residual at the point M are determined according to the fitting plane.

[0191] All related contents of each step involved in the foregoing embodiments of the point cloud segmentation method can be cited to the function description of the function modules corresponding to the point cloud segmentation device in the embodiments of the application, and will not be repeated here.

[0192] The division of the modules in the embodiments of the application is illustrative, and is merely a logical function division. In actual implementation, another division mode can be used. In addition, each function module in each embodiment of the application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0193] Based on the same inventive concept, the embodiments of the application provide an electronic device. Please refer to Figure 8 The electronic device includes at least one processor 801 and a memory 802 connected with the at least one processor. In the embodiments of the application, the specific connection medium between the processor 801 and the memory 802 is not limited, Figure 3The connection between the processor 801 and the memory 802 is taken as an example, and the bus 800 is connected between Figure 3 The connection between the processor 801 and the memory 802 is taken as an example, and the bus 800 is connected between Figure 3 The connection between the processor 801 and the memory 802 is taken as an example, and the bus 800 is connected between

[0194] In the embodiments of the present application, the memory 802 stores instructions executable by the at least one processor 801, and the at least one processor 801 can execute the steps included in the point cloud segmentation method described above by executing the instructions stored in the memory 802.

[0195] The processor 801 is the control center of the electronic device, and can connect various parts of the electronic device through various interfaces and lines. By running or executing the instructions stored in the memory 802 and calling the data stored in the memory 802, the electronic device can process various functions and data, thereby monitoring the whole electronic device. Optionally, the processor 801 can include one or more processing units, and the processor 801 can integrate an application processor and a modem processor. The application processor mainly processes operating systems and application programs, and the modem processor mainly processes wireless communication. It can be understood that the modem processor can also not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0196] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the point cloud segmentation method disclosed in the embodiments of the present application can be directly embodied as execution by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0197] The memory 802, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 802 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 802 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 802 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0198] By designing and programming the processor 801, the code corresponding to the point cloud segmentation method introduced in the foregoing embodiments can be fixed in the chip, so that the chip can execute the steps of the foregoing point cloud segmentation method at runtime. How to design and program the processor 801 is a technology known to those skilled in the art, and will not be described here.

[0199] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium storing computer instructions, when the computer instructions run on a computer, the computer instructions make the computer execute the steps of the point cloud segmentation method as described above.

[0200] In some possible implementations, various aspects of the point cloud segmentation method provided by the present application can also be implemented in the form of a program product, which includes program codes, when the program product runs on an electronic device, the program codes are used to make the detection device execute the steps in the point cloud segmentation method according to various exemplary embodiments of the present application described in the specification.

[0201] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0202] The present application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems) and computer program products according to this application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing machine, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0203] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0205] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their legal equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for point cloud segmentation, characterized in that, The method comprises: acquiring point cloud data of a target region; preprocessing the point cloud data to obtain target point cloud; wherein the number of points in the target point cloud is less than the number of points in the target region; calculating the normal vector and the residual of each point after preprocessing; wherein the normal vector is a vector represented by a straight line perpendicular to the plane where the point is located, and the residual is used to indicate the curvature of the surface where each point is located; plane segmentation is performed on the target region according to the normal vector and the residual of each point; wherein the preprocessing of the point cloud data comprises: mean filtering and median filtering are performed on the point cloud data, specifically: mean filtering is performed on the point cloud data to obtain the first coordinate value of each point, and median filtering is performed on the point cloud data to obtain the second coordinate value of each point; the first coordinate value and the second coordinate value of the same point are compared; the smaller coordinate value is determined as the coordinate value of the corresponding point after filtering.

2. The method of claim 1, wherein, The preprocessing of the point cloud data further comprises: down-sampling processing is performed on the point cloud data to obtain first point cloud data; outlier noise in the first point cloud data is filtered out to obtain second point cloud data; mean filtering and median filtering are performed on the second point cloud data to obtain third point cloud data; erosion operation is performed on the third point cloud data to obtain the point cloud data after preprocessing.

3. The method of claim 2, wherein, The filtering out of the outlier noise in the first point cloud data comprises: the density of the first point cloud data is calculated; the point cloud with a density lower than a first preset threshold is filtered out.

4. The method of claim 1, wherein, The plane segmentation of the target region according to the normal vector and the residual of each point comprises: a first seed point Q1 with the minimum residual is determined from the target point cloud; points satisfying a preset condition are determined from the target point cloud with the first seed point Q1 as the center, the point Q1 and the points satisfying the preset condition form a first plane, and the first plane is the first plane segmented from the target region; a second seed point Q2 is determined in the remaining point cloud; wherein the remaining point cloud is the point cloud in the target point cloud except the points contained in the first plane; when the number of remaining points in the target point cloud is less than a second preset threshold, the plane segmentation is stopped.

5. The method of claim 4, wherein, The determination of the points satisfying the preset condition from the target point cloud comprises: the included angle between the normal vector of the other points in a first region except the point Q1 and the normal vector of the point Q1 is calculated; wherein the first region is a circular region with the point Q1 as the center and a preset size as the radius; the points with an included angle less than a preset angle threshold are determined as the points satisfying the preset condition.

6. The method of claim 5, wherein, Further comprising: convexity of each point N in the points with an included angle less than a preset angle threshold is determined, the convexity is used to indicate whether the surface where the point N is located is a convex surface or a concave surface; the points with a convexity determination result are determined as the points in the same plane as the point Q1 to obtain the first plane.

7. The method of claim 6, wherein, The convexity determination of each point N in the points with an included angle less than a preset angle threshold comprises: convexity determination is performed through the following formula: wherein and are the normal vectors to the point Q1 and to the point N, respectively, and is the directional vector from the point N to the point Q1. In the method When the value is greater than a third preset threshold, the point N is determined as a convex point. When the value is less than the third preset threshold, the point N is determined as a concave point.

8. The method of claim 1, wherein, The calculation of the normal vector and the residual of each point after preprocessing comprises: determine a fitting plane at the point M after the preprocessing by a least square method; determine a normal vector and a residual error at the point M according to the fitting plane.

9. An apparatus for point cloud segmentation, the apparatus comprising: The device comprises: an acquisition module configured to acquire point cloud data of a target region; a processing module configured to preprocess the point cloud data to obtain target point cloud, wherein a number of points in the target point cloud is less than a number of points in the target region; the processing module is further configured to calculate a normal vector and a residual error of each point after the preprocessing, wherein the normal vector is a vector represented by a straight line perpendicular to a plane where the point is located, and the residual error is used to indicate a curvature of a surface where the point is located; the processing module is further configured to perform plane segmentation on the target region according to the normal vector and the residual error of each point, wherein the processing module is specifically configured to: perform mean filtering and median filtering on the point cloud data, specifically, perform mean filtering on the point cloud data to obtain a first coordinate value of each point, and perform median filtering on the point cloud data to obtain a second coordinate value of each point; compare the first coordinate value and the second coordinate value of the same point; determine a smaller coordinate value as a coordinate value of the corresponding point after filtering.

10. The apparatus of claim 9, wherein, The preprocessing of the point cloud data by the processing module further comprises: perform down-sampling processing on the point cloud data to obtain first point cloud data; filter out outliers in the first point cloud data to obtain second point cloud data; perform mean filtering and median filtering on the second point cloud data to obtain third point cloud data; perform an erosion operation on the third point cloud data to obtain point cloud data after the preprocessing.

11. The apparatus of claim 10, wherein, The processing module is specifically configured to: calculate a density of the first point cloud data; filter out point cloud with a density lower than a first preset threshold.

12. The apparatus of claim 9, wherein, The processing module is specifically configured to: determine a first seed point Q1 with a minimum residual error from the target point cloud; determine points satisfying a preset condition from the target point cloud with the first seed point Q1 as a center, the point Q1 and the points satisfying the preset condition form a first plane, and the first plane is a first plane segmented from the target region; determine a second seed point Q2 from remaining point cloud, wherein the remaining point cloud is point cloud excluding points included in the first plane from the target point cloud; stop the plane segmentation when a number of remaining points in the target point cloud is less than a second preset threshold.

13. The apparatus of claim 12, wherein, The processing module is specifically configured to: calculate an included angle between a normal vector of a point other than the point Q1 in a first region and a normal vector of the point Q1, wherein the first region is a circular region with the point Q1 as a center and a preset size as a radius; determine a point with an included angle less than a preset angle threshold as a point satisfying the preset condition.

14. The apparatus of claim 13, wherein, The processing module is further configured to: perform concave-convex judgment on each point N from the points with the included angle less than the preset angle threshold, the concave-convex judgment is used to indicate whether a surface where the point N is located is a convex surface or a concave surface; determine a point with a judgment result of convexity as a point in the same plane as the point Q1 to obtain the first plane.

15. The apparatus of claim 14, wherein, The processing module is specifically configured to: perform the concave-convex judgment by the following formula: wherein and are the normal vectors to the point Q1 and to the point N, respectively, and is the directional vector from the point N to the point Q1. In the method When the value is greater than a third preset threshold, the point N is determined as a convex point, and when the value is less than the third preset threshold, the point N is determined as a concave point. When the value is greater than a third preset threshold, the point N is determined as a convex point, and when the value is less than the third preset threshold, the point N is determined as a concave point.

16. The apparatus of claim 9, wherein, The processing module is specifically configured to: determining a fitting plane at the point M after the pre-processing by least square method; determining a normal vector and a residual at the point M according to the fitting plane.

17. An electronic device, comprising: comprising: a memory for storing program instructions; a processor for invoking the program instructions stored in the memory and executing the steps included in the method according to the obtained program instructions.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program includes program instructions, which, when executed by a computer, cause the computer to execute the method according to any one of claims 1-8.

Images