Method, device and computer readable medium for segmenting point cloud data

Abstract

The application discloses a method, device and equipment for segmenting point cloud data and a computer readable medium, and relates to the technical field of computers. A specific embodiment of the method comprises: dividing point cloud data into patches of the point cloud data, and assigning semantic labels to the patches of the point cloud data; merging patches with the same semantic label into the same object instance according to a patch merging condition and a repulsive patch merging condition, wherein the patch merging condition and the repulsive patch merging condition are extracted from a cluster group of training data; and segmenting the point cloud data according to the object instance. The embodiment can shorten the time for segmenting point cloud data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a method, device and equipment for segmenting point cloud data and a computer readable medium. BACKGROUND

[0002] Geometry-based shape segmentation is an important means of three-dimensional data analysis, and has a wide range of applications in target detection, scene modeling, scene understanding and other research fields. Patch segmentation is a main technical means of point cloud geometry shape analysis. Reasonable patch segmentation results can effectively classify point sets, and help to extract geometry shape features and generate semantic information.

[0003] In the process of implementing the present application, the inventors have found that at least the following problems exist in the prior art: In actual applications, the traditional region growing algorithm can lead to unstable patch segmentation. In order to avoid patch ambiguity caused by under-segmentation, a strict region growing parameter is used to over-segment the scene. Patch ambiguity refers to vertices in the same patch containing different semantic labels. Patch over-segmentation leads to a long time spent in segmenting point cloud data when facing large-scale three-dimensional scenes with high patch complexity. SUMMARY

[0004] Therefore, the embodiments of the present application provide a method, device and equipment for segmenting point cloud data and a computer readable medium, which can shorten the time for segmenting point cloud data.

[0005] To achieve the above object, according to one aspect of the embodiments of the present application, a method for segmenting point cloud data is provided, comprising:

[0006] segmenting point cloud data into patches of point cloud data, and assigning semantic labels to the patches of point cloud data;

[0007] merging patches with the same semantic label into the same object instance according to patch merging conditions and repulsive patch merging conditions, wherein the patch merging conditions and the repulsive patch merging conditions are extracted from a cluster group of training data;

[0008] segmenting the point cloud data according to the object instance.

[0009] The method further comprises:

[0010] segmenting training data into training patches according to training semantic labels and training instance labels, wherein the training semantic labels are used to identify statistical features of object instances, and the training instance labels are used to identify geometric features of object instances;

[0011] dividing the training patches into the patch cluster groups by using a clustering algorithm, so as to extract the patch merging conditions and the repulsive patch merging conditions in each patch cluster group.

[0012] The extracting the merging condition and the excluding merging condition in each patch cluster group comprises:

[0013] According to the oriented bounding box of the training patch, the spatial relationship of the two training patches as a patch pair is determined;

[0014] According to the spatial relationship of the patch pair, the cluster group of the patch pair and the training semantic label of the patch pair, the appearance frequency of the patch pair and the appearance proportion of the patch pair are extracted;

[0015] According to the appearance frequency and the appearance proportion, the merging condition and the excluding merging condition are determined.

[0016] The point cloud data is divided into patches of point cloud data, and a semantic label is assigned to each patch of point cloud data, comprising:

[0017] The point cloud data is divided into patches of point cloud data;

[0018] The semantic confidence of the patch of point cloud data is calculated;

[0019] The maximum value of the semantic confidence corresponds to the semantic label, which is taken as the semantic label of the patch of point cloud data.

[0020] The point cloud data is divided into patches of point cloud data, comprising:

[0021] The point cloud data is divided into multiple levels, and the point cloud data at the current level is divided into multiple patches;

[0022] If the divided patch does not meet the stop dividing condition, the divided patch is divided again at the next level;

[0023] If the divided patch meets the stop dividing condition, the patch meeting the stop dividing condition is taken as the patch of point cloud data.

[0024] The stop dividing condition comprises:

[0025] The semantic label of the divided patch is different from the semantic label of the undivided patch, and the confidence of the divided patch is greater than the confidence of the undivided patch.

[0026] The patches with the same semantic label are merged into the same object instance according to the merging condition and the excluding merging condition, comprising:

[0027] A patch pair is constructed with two patches with the same semantic label;

[0028] The patch pair is merged into one object instance if the patches in the patch pair belong to the same object instance, the patch pair meets the patch merging rule, and the patch pair does not meet the exclusion patch merging rule.

[0029] According to a second aspect of the embodiments of the present application, a device for segmenting point cloud data is provided, comprising:

[0030] a label module configured to split the point cloud data into patches of the point cloud data, and assign semantic labels to the patches of the point cloud data;

[0031] a merging module configured to merge the patches with the same semantic labels into one object instance according to a patch merging condition and an exclusion patch merging condition, the patch merging condition and the exclusion patch merging condition being extracted from a cluster group of training data;

[0032] a segmentation module configured to segment the point cloud data according to the object instance.

[0033] According to a third aspect of the embodiments of the present application, an electronic device for segmenting point cloud data is provided, comprising:

[0034] one or more processors;

[0035] a storage configured to store one or more programs,

[0036] when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0037] According to a fourth aspect of the embodiments of the present application, a computer readable medium having a computer program stored thereon is provided, the program being executed by a processor to implement the method as described above.

[0038] One of the embodiments of the above application has the following advantages or beneficial effects: the point cloud data is split into patches of the point cloud data, and semantic labels are assigned to the patches of the point cloud data; the patches with the same semantic labels are merged into one object instance according to a patch merging condition and an exclusion patch merging condition, the patch merging condition and the exclusion patch merging condition being extracted from a cluster group of training data; and the point cloud data is segmented according to the object instance. The patches belonging to the same object instance are merged by using the patch merging condition and the exclusion patch merging condition, and finally the point cloud data is segmented into multiple object instances, so as to avoid under-segmentation and over-segmentation, and the time for segmenting the point cloud data can be shortened.

[0039] The further effects of the above non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings are used to better understand the present application, and do not constitute undue limitations on the present application. Among them:

[0041] Figure 1 is a schematic diagram of the main process of the method for segmenting point cloud data according to an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of the method architecture for segmenting point cloud data according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the process of obtaining a patch merging condition and an exclusion patch merging condition according to an embodiment of the present application;

[0044] Figure 4 is a schematic diagram of the process of segmenting training point cloud data according to an embodiment of the present application;

[0045] Figure 5 is a schematic diagram of the process of dividing a patch into a patch clustering group according to an embodiment of the present application;

[0046] Figure 6 is a schematic diagram of the process of extracting a context rule in a patch clustering group according to an embodiment of the present application;

[0047] Figure 7 is a schematic diagram of the process of dividing point cloud data into patches of point cloud data according to an embodiment of the present application;

[0048] Figure 8 is a schematic diagram of the process of assigning a semantic label to a patch according to an embodiment of the present application;

[0049] Figure 9 is a schematic diagram of the process of merging patches into the same object instance according to an embodiment of the present application;

[0050] Figure 10 is a schematic diagram of the main structure of the device for segmenting point cloud data according to an embodiment of the present application;

[0051] Figure 11 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0052] Figure 12 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0053] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the application. Accordingly, one skilled in the art should realize that various changes and modifications in the preferred embodiments described herein can be made without departing from the scope and spirit of the application. Also, for clarity and the sake of brevity, the detailed description is not meant to be exhaustive or limit the application to the precise form disclosed. Many modifications and variations are possible in light of this disclosure.

[0054] Conventional point cloud segmentation methods mainly analyze the geometric shape information of point cloud itself through common mathematical geometric models. For example, a typical geometric primitive is explicitly fitted to the input point cloud to achieve primitive segmentation. The geometric primitives include representative geometric bodies, such as cylinders, spheres, and cones.

[0055] However, real point cloud scenes often have irregular noise and complex deformation. Therefore, a large geometric primitive library is needed to support the adaptation process. Objects can be segmented into multiple components using convexity cues and shape labels. Efficient RANSAC algorithms can be used for shape segmentation. Hierarchical analysis can be performed on sampled surface point clouds to achieve segmentation.

[0056] The above methods are based on geometric shape analysis and do not consider semantic information, which can cause unstable primitive segmentation boundaries and other problems. In addition, the above segmentation methods often require frequent user interaction to obtain high-quality segmentation results.

[0057] Three-dimensional mesh is another representation of a three-dimensional scene. The goal of mesh segmentation is to divide the complete three-dimensional object into meaningful geometric components. Learning methods can be used to control mesh segmentation using semantic information and have achieved significant improvements. Collaborative analysis can be used to improve mesh segmentation and prove the effectiveness of building a "shape-component" statistical model. This method uses unsupervised learning and only divides objects into multiple components based on geometric similarity. However, its application is limited to high-quality three-dimensional meshes with consistent categories and similar component compositions.

[0058] Three-dimensional mesh segmentation is difficult to directly extend to complex three-dimensional point cloud scenes. This is because scene point clouds often have a lot of noise and occlusions, flexible object layouts, and complex spatial relationships. In addition, each object category can have different geometric shapes and diverse component compositions.

[0059] Shape segmentation greatly inspires point cloud semantic segmentation. Some methods treat the segmenting and semantic labeling as two independent problems, and use traditional region growing algorithm to segment the point cloud. Based on the segmenting, color, segment geometry and camera parameters are taken as input, and then support vector machine model is used to learn object semantic information and context information between segments. In the prediction process, segment semantic label is formulated as an optimization problem, and the method is solved by mixed integer programming.

[0060] From the practice of segmenting point cloud data, segments of the same object usually have various shape changes, such as different shapes of chair backs. Segments of similar shapes can also belong to different parts of objects, such as erasers and telephone microphones. The above cases will cause long time consumption in segmenting point cloud data.

[0061] In order to solve the problem of long time consumption in segmenting point cloud data, the technical solutions in the embodiments of the present application can be used.

[0062] Referring to Figure 1 , Figure 1 The method for segmenting point cloud data according to the embodiment of the present application mainly includes the following steps: merging segments of the same object instance according to segment merging conditions and excluding segment merging conditions, as shown in Figure 1 .

[0063] In the embodiment of the present application, three-dimensional point cloud data is input, each vertex contains coordinate position information: P = (p x ,p y ,p z ) and normal information: n = (n x ,n y ,n z ). Each vertex in the training cloud data set has a semantic label (label) and an instance label (obj). The semantic label is used to identify the statistical characteristics of the object instance, and the instance label is used to identify the geometric characteristics of the object instance.

[0064] The object of the embodiment of the present application is to assign a semantic label to each vertex to realize semantic segmentation, and to fuse segments into object instances according to the semantic labels of the segments. It should be noted that the embodiment of the present application focuses on analyzing geometric information and ignoring color information. Although color is very helpful in a small range of similar scenes, color cannot be directly used as a semantic feature in a large-scale scene. Moreover, the color information of the vertices of the point cloud data is easily affected by factors such as weather, light and acquisition equipment.

[0065] Referring to Figure 2 ,Figure 2 is a schematic diagram of a method for segmenting point cloud data according to an embodiment of the present application. Figure 2 The method comprises two stages, a training stage and a testing stage.

[0066] First, in the training stage, the training data is segmented into patches using known training semantic labels and training instance labels. Making full use of the training instance labels can ensure that the object boundaries are well preserved, and the segmentation results are optimized to avoid under-segmentation and reduce over-segmentation. Then, in order to summarize the geometric similarity of the patches, a clustering algorithm is used to group the patches. Using the clustering group results as the intermediate representation of the data, the context rules can be more reliably analyzed and extracted. The context rules include patch merging conditions and repulsive patch merging conditions.

[0067] In the testing stage, the patches generated from different segmentation levels are classified according to the learned context rules to determine the reasonable segmentation level. Finally, the merging of the patches is guided by the patch-instance relationship and the semantic labels of the patches to achieve instance segmentation.

[0068] S101, segmenting point cloud data into patches of point cloud data, and assigning semantic labels to the patches of point cloud data.

[0069] Before segmenting the point cloud data, the context rules of the training data need to be obtained, i.e., the patch merging conditions and the repulsive patch merging conditions.

[0070] Referring to Figure 3 , Figure 3 is a flowchart of obtaining patch merging conditions and repulsive patch merging conditions according to an embodiment of the present application. Specifically, the following steps are included:

[0071] S301, segmenting training data into training patches according to training semantic labels and training instance labels, the training semantic labels being used to identify the statistical features of object instances, and the training instance labels being used to identify the geometric features of object instances.

[0072] Patch segmentation is the basis for subsequent geometric feature learning. The traditional region growing algorithm is determined by random seed point selection and boundary control parameters. If the same segmentation parameters are used for patch segmentation of different objects, there is a high probability that different degrees of over-segmentation or under-segmentation will occur.

[0073] Over-segmentation can effectively separate the boundaries of different objects, but it will cause the patch segmentation results to be excessively fragmented and unable to maintain the geometric shape. Under-segmentation will cause the object boundaries to not be correctly separated, resulting in the vertices within a single patch not having a unified semantic label. The main reason for these phenomena is that the geometric structures of different types of objects are inconsistent. Of course, the normal noise of the point cloud is also a key inducement for segmentation errors.

[0074] For normal noise, the embodiment of the present application proposes a lightweight dynamic region growing algorithm, which effectively divides the point cloud into a set of mutually disjoint and smooth shape planar patches. Unlike traditional region growing algorithms, the dynamic region growing algorithm dynamically updates the shape features of the patch during region growing, which is more adaptive to normal noise and point cloud sampling.

[0075] In the embodiment of the present application, the training data is specifically training point cloud data. By adopting the embodiment in Figure 4 , the training point cloud data is segmented.

[0076] Referring to Figure 4 , Figure 4 is a flowchart of segmenting training point cloud data according to the embodiment of the present application, specifically comprising the following steps:

[0077] S401, estimate the curvature of the training point cloud data, the curvature is determined by the ratio of the minimum eigenvalue in the vertex set to the sum of all eigenvalues in the vertex set, and the vertex set includes the vertex set included in the neighborhood of each vertex.

[0078] In the embodiment of the present application, the curvature of each training point cloud data can be estimated using the definition in the Point Cloud Library (PCL). Unlike differential geometry, the curvature of the training point cloud data is defined as the ratio of the minimum eigenvalue generated by principal component analysis to the sum of all eigenvalues, which is also an indicator describing the bending degree of a plane. Here, the object of principal component analysis is the vertex set included in the neighborhood of each vertex. As an example, the neighborhood radius is set to 2 centimeters.

[0079] That is, the curvature is determined by the ratio of the minimum eigenvalue in the vertex set to the sum of all eigenvalues in the vertex set, and the vertex set includes the vertex set included in the neighborhood of each vertex.

[0080] S402, select the current curvature minimum point as the seed point of region growing.

[0081] For each patch segmentation process, the current curvature minimum point is selected as the seed point of region growing. Wherein, the current curvature minimum point is the noise minimum point.

[0082] S403, in the patch where the seed point is located, the vertex satisfying the constraint condition is added to the patch, and the constraint condition includes that the newly added vertex and the seed point are in a plane, the newly added vertex and the seed point have the same label, and the label includes the training semantic label and / or the training instance label.

[0083] In the patch growing process, considering any known vertex p(x)∈P i , the unknown vertex p(y) is allowed to be added to Pi sufficient condition in Formula 1:

[0084]

[0085] wherein constant a1 represents the radius of region growing. As an example, a1 is fixedly set as 1 centimeter. Constant a2 represents the coplanarity constraint of the growing process. As an example, a2 is fixedly set as 0.5 centimeter. n(y) represents the normal information of the growing vertex p(y). i represents the barycenter of the patch. n i represents the normal of the patch P i . Formula 1 guarantees the continuity, smoothness and planarity of the patch as a whole during the region growing process.

[0086] Exemplarily, Formula 1 is summarized as constraint conditions. The constraint conditions include that the newly added vertex and the seed point are on a same plane, the newly added vertex and the seed point have the same label, and the label includes the training semantic label and / or the training instance label.

[0087] In an embodiment of the present application, it is worth noting that the barycenter and the normal of the patch will dynamically change during the region growing process of the patch. However, it is unreasonable to re-evaluate the patch, i.e. to re-calculate the barycenter and the normal of the patch, every time a vertex is added, which will bring about excessively high computational complexity. Thus, the barycenter and the normal of the patch in which the number of vertices is greater than a preset vertex threshold are updated.

[0088] Specifically, the barycenter and the normal of the patch are re-calculated only when the number of vertices |P i | of the patch is increased to . The preset vertex threshold is wherein is the number of points in the last patch evaluation. a3 is a constant factor, and as an example, a3 is fixedly set as 1.5. Thus, the number of times of patch evaluation during the region growing process will not exceed The overall algorithm has a complexity of O(NlogN).

[0089] In an embodiment of the present application, k1 and k2 determine the boundary of the region growing, and are key parameters for controlling the segmentation result of the patch. A loose parameter setting can cause an under-segmentation phenomenon of the patch. Similarly, a strict constraint condition can cause an over-segmentation phenomenon of the patch. Since the under-segmentation phenomenon causes different objects to be mixed in the same patch, which causes objective ambiguity for semantic classification, most of the existing methods usually tend to select an over-segmentation parameter.

[0090] As an example, in the training phase of the embodiments of the present application, k1 and k2 are set to 0.5 cm and 5 cm respectively, and the constraint condition of the region growing algorithm requires that the patches belong to the same instance during the entire patch growing process. That is, the newly added vertex has the same label as the seed point, and the label includes the training semantic label and / or the training instance label. For example, when the training instance label does not exist, it is required that the patches have the same training semantic label

[0091] In Figure 4 In the embodiments, the vertex satisfying the constraint condition is added to the patch. Compared with the over-segmentation phenomenon caused by introducing noise, the scheme in the embodiments of the present application largely avoids normal noise, has stronger adaptability to noise, and suppresses the over-segmentation phenomenon.

[0092] S302, using a clustering algorithm to divide the patches into patch clustering groups, to extract patch merging conditions and repelling patch merging conditions in each patch clustering group.

[0093] After the training point cloud data is segmented into patches, in order to improve the efficiency of the training semantic label, based on the similarity of the geometric shapes of the patches, an unsupervised clustering is used for simple shape grouping. The clustering group result is an important intermediate for "shape-semantic" relationship learning and context analysis.

[0094] Referring to Figure 5 , Figure 5 is a flowchart of the division of the patches into patch clustering groups according to the embodiments of the present application, and specifically includes the following steps:

[0095] S501, calculating the oriented bounding box of each patch to fit the corresponding rectangular patch.

[0096] In order to better cluster the patches according to the geometric shapes, it is necessary to define the geometric shape descriptors. In the embodiments of the present application, the oriented bounding box (OBB) of each patch is first calculated.

[0097] Specifically, the oriented bounding box takes the normal of the patch as the main direction, and projects the patch point set to the fitting tangent plane according to the normal. Then, the fitting rectangle is calculated in the fitting plane. The fitting rectangle is the smallest rectangle that encloses all the point sets.

[0098] As an example, the convex hull algorithm and the rotating calipers algorithm are used to effectively obtain the optimal solution. The convex hull algorithm has a time complexity of O(nlogn), and the rotating calipers algorithm has a time complexity of 0(n), where n is the number of vertices in the patch.

[0099] As an example, the segmentation result of a chair with a backrest includes 6 oriented bounding boxes. Respectively, the oriented bounding box of the backrest, the oriented bounding box of the seat of the chair, and the four oriented bounding boxes corresponding to the four supporting legs.

[0100] An oriented bounding box is a good shape approximation, which can efficiently assist the extraction of patch features and the calculation of patch relationships. As an example, around an oriented bounding box, 9 feature descriptors are defined, and the specific meanings are shown in Table 1.

[0101] Table 1

[0102]

[0103] The feature descriptors of the patches in Table 1. The concave hull refers to the smallest concave polygon covering all the vertices of the patch. Compared with the convex hull, the concave hull can more accurately describe the surface area of a complex patch, such as an L-shaped table top. These feature descriptors cover the position, shape, size, and distribution of a given patch, are easy to calculate, and are generally applicable.

[0104] S502, divide the rectangular corresponding patch into a patch cluster group through the improved K-means++ algorithm. The number of clusters in the improved K-means++ algorithm is dynamically determined according to the maximum patch distance constant.

[0105] In the embodiment of the present application, patch clustering is introduced as an intermediate representation, which bridges the gap between patches and objects. Clustering can reliably summarize the geometric shape descriptors of patches and reliably learn context rules from instance object distributions.

[0106] Referring to Table 1, each patch P i corresponds to a 9-dimensional feature Since the same dimension of feature values has a large variation, for example, the height of an object on the ground relative to the floor can be close to zero, but relative to the ceiling it is a larger value. In the embodiment of the present application, no clustering is directly performed in this feature space.

[0107] The variation range of different features also has a large difference, for example, the value range of the angle is very different from the value range of the height. In order to solve the above problems, the standard deviation σ k of the feature is used to normalize each feature and use logarithmic operation to handle the large difference in the value.

[0108] Specifically, the converted feature vector is calculated by the following formula

[0109] where ε is a small offset to ensure the validity of the In function. In S502, ε is set to 1 / e, e is a natural constant.

[0110] The distance between the patches P x and P y is defined as the l1 norm between the feature vectors V x and V y :

[0111]

[0112] In the embodiment of the present application, the improved K-means++ algorithm is used to perform fast and effective clustering. The K-means++ algorithm is a fast and efficient clustering algorithm, which can quickly converge through the specified K good initial values of the clustering center.

[0113] The time complexity of selecting the clustering center is O(KN), where N is the number of patches, K is the number of clusters, and generally K<N. The original K-means++ algorithm needs to specify the number of clusters K, and this value has a decisive influence on the clustering result.

[0114] In geometric shape clustering, it is often difficult to make a reasonable prior assumption about the number of groups. In the embodiment of the present application, the number of clusters K is not fixed in advance, but is adaptively determined by defining a maximum patch distance constant a4. That is, the number of clusters in the improved K-means++ algorithm is dynamically determined according to the maximum patch distance constant.

[0115] The improved K-means++ algorithm includes the following 6 steps, a4 represents the maximum patch distance that cannot be allocated to the same cluster, and its value is highly related to the geometric features. As an example, a4 is fixed at 1.7, which has good adaptability to different data sets.

[0116] (1) Randomly select a sample from the patch set features and take it as the clustering center, let K=1.

[0117] (2) Calculate the distance between each patch feature V i and the nearest clustering center

[0118] (3) Let represent the feature with the farthest distance from the clustering center among all features.

[0119] (4) Add to the cluster, let K=K+1.

[0120] (5) Repeat (2) until

[0121] (6) Use K cluster centers as initial values ​​to perform K-Means clustering.

[0122] After clustering using the improved K-means++ algorithm, each fragment P i Each fragment P in the training set is assigned to the nearest cluster center. i It also includes semantic tags for object categories. i .

[0123] In this embodiment of the invention, statistical information from the training set is used to pre-calculate the training semantic labels l. i The basic confidence level between cluster c and cluster c is conf base and the average eigenvector of each cluster. Cluster center:

[0124]

[0125]

[0126] Where |.| is the number of elements in the set, and mean{.} is the vector mean of the elements in the set.

[0127] exist Figure 5 In the embodiment, the number of clusters in the improved K-means++ algorithm is dynamically determined based on the maximum fragment distance constant, thereby improving the accuracy of fragment clustering.

[0128] Fragment clustering results can provide some guidance for semantic recognition. However, many fragments may have similar shapes but inconsistent semantics, and the clustering of fragment shapes cannot directly determine the training semantic labels for objects. Generally, contextual information analysis focuses primarily on the object instance level. This is highly sensitive to the accuracy of segmentation and requires explicit modeling of the object and its components.

[0129] See Figure 6 , Figure 6 This is a flowchart illustrating the extraction of context rules from each fragment cluster group according to an embodiment of the present invention, specifically including the following steps:

[0130] S601. Determine the spatial relationship between two fragments forming a fragment pair based on the directed bounding box of the fragment.

[0131] In the embodiments of the present application, the context rules are defined on the basis of the spatial distribution and geometric relationship of "patch pairs", the relationship between the clustered combination of patch pairs and their semantic combination is counted, and then the regular combination is summarized and extracted. The context rules imply the context relationship between "part-part", "part-object" and "object-object". In order to count the geometric relationship of all "patch pairs", it is necessary to quickly and efficiently judge the proximity relationship between patches, that is, to calculate the distance between two patches P x and P y . In order to simplify the calculation of distance, the nearest distance d(P x , P y ) between the corresponding oriented bounding boxes of the two patches is used to approximate replacement, which is specifically defined as follows:

[0132]

[0133] Where OBB X and OBB y represent the point set inside the oriented bounding box, and X and Y are any points therein. In order to simplify the calculation, it is first checked whether If there is an intersection, d(OBB X , OBB y ) is defined as 0, otherwise d(OBB X , OBB y ) can be simplified to:

[0134]

[0135] Where V and S represent the vertices and surfaces of the corresponding oriented bounding box.

[0136] In the embodiments of the present application, the adjacent distance threshold is set to d1=0.50 meters, and the definition of adjacent patch pair P is:

[0137] P={(P x , P y )|d(OBB x , OBB y )≤d1} Formula 7

[0138] For any patch pair (P x , P y ) ∈ P, it is necessary to further analyze their spatial relationship r xy . The spatial relationship includes one or more of the following possibilities: adjacent, that is, the distance between patches is less than 0.5 meters; close, that is, the distance between patches is less than 0.03 meters; perpendicular; parallel and coplanar.

[0139] S602, extract the occurrence frequency of the fragment pair and the occurrence proportion of the fragment pair according to the spatial relationship of the fragment pair, the cluster group of the fragment pair and the semantic label of the fragment pair.

[0140] For each fragment pair (P x ,P y )∈P in the training set, a five-tuple q=(c x ,c y ,r xy ,l x ,l y ) can be used to describe the context. Wherein c x and c y represent the cluster group number corresponding to the fragment pair, r xy is the spatial relationship described above, l x and l y are the semantic labels of the fragments.

[0141] In an embodiment of the present application, the occurrence frequency of different five-tuples is counted, denoted as γ q . Among all the fragment pairs in the training set belonging to the same cluster c x and c y , and having the same relationship r xy with q, the occurrence proportion with the same semantic label is further calculated, denoted as τ q :

[0142]

[0143] S603, determine the fragment merging rule and the exclusion fragment merging rule according to the occurrence frequency and the occurrence proportion.

[0144] Here, the set of fragment context five-tuples is denoted as Q, and the sufficient and necessary condition for q∈Q is:

[0145] γ q ≥t1 Formula 9

[0146] τ q ≥t2 Formula 10

[0147] Formula 9 restricts that the five-tuple should occur a sufficient number of times to filter out uncommon combinations. Formula 10 restricts the strength of the context rule, that is, the context relationship formed by the five-tuple should have a dominant level of proportion in the same cluster combination. As an example, t1 is set to 2 and t2 is set to 75%.

[0148] In an embodiment of the present application, a four-tuple t=(c x ,c y ,r′ xy ,l) is defined to describe the "cluster-instance" relationship of the fragment. γ′(cx ,c y ,r′ xy ,l,1) represents the frequency of occurrence of the merged quintuple; τ'(c x ,c y ,r′ xy ,l,1) represents the proportion of the merged quintuple that supports the same instance relationship of the same object.

[0149] Let τ+ represent the positive "shard-instance" contextual relationship, indicating support for the merging operation between shards. A sufficient and necessary condition for t e τ+ to hold is that the shard merging rule is as follows:

[0150] γ'(c x ,c y ,r′ xy ,l,1) > t3 Equation 11

[0151] τ'(c x ,c y ,r′ xy ,l,1) > t4 Equation 12.

[0152] Similarly, γ'(c x ,c y ,r′ xy ,l,0) represents the frequency of occurrence of the merged quintuple; τ'(c x ,c y ,r′ xy ,l,0) represents the proportion of the merged quintuple that does not support the same instance relationship of the same object.

[0153] Let τ- represent the negative "shard-instance" contextual relationship, indicating exclusion of the merging operation between shards. A sufficient and necessary condition for t e τ- to hold is that the exclusion shard merging rule is as follows:

[0154] γ'(c x ,c y ,r′ xy ,l,0) > t3 Equation 13

[0155] τ'(c x ,c y ,r′ xy ,l,0) > t4 Equation 14

[0156] where t3 constrains the minimum frequency of occurrence of the quintuple; t4 constrains the proportion of the instance relationship in all relevant cluster combinations, i.e., it indicates that the majority of similar shard combinations in the training set belong to the same / different object instance. As an example, t3 is set to 2 and t4 is set to 90%.

[0157] In Figure 6In this embodiment, the rules for merging fragment clouds and the rules for excluding fragment merging are learned by training point cloud data, thereby laying the foundation for merging fragments.

[0158] For point cloud data, the goal of semantic segmentation is to divide it into fragments of point cloud data and assign the correct semantic label to each fragment.

[0159] See Figure 7 , Figure 7 This is a flowchart illustrating the process of segmenting point cloud data into point cloud data fragments according to an embodiment of the present invention, specifically including the following steps:

[0160] S701. Divide the point cloud data into multiple levels, and divide the point cloud data into multiple fragments according to the current level.

[0161] Fragment-based segmentation should generate a consistent set of fragments for both training and testing phases, so that the learned model can be applied effectively.

[0162] However, this is controlled by parameters k1 and k2. During the training phase, prior training semantic labels and / or training instance labels help guide segmentation. Therefore, by setting more relaxed parameters, the resulting fragment results can largely avoid undersegmentation or oversegmentation.

[0163] However, during the testing phase, specifically the segmentation of point cloud data, there is objectively no single set of parameters k1 and k2 that works in all situations. For example, using lenient k1 and k2 cannot segment a blackboard from a wall because the blackboard and the wall have similar depths and normals. Conversely, using stricter k1 and k2 would oversegment complex curved surfaces like sofas into numerous tiny fragments, resulting in the loss of geometric shape.

[0164] To reduce the occurrences of undersegmentation and oversegmentation, appropriate local control parameters can be selected through top-down enumeration.

[0165] Semantic labels require a series of segmentation scale levels, each level being determined by k1 and k2 corresponding to that level, defined as: As an example, for each level, the segmentation step size can be determined based on k1 and k2 corresponding to that level. Here, k1 and k2 for each level are preset parameters.

[0166] First, initialize level 0 by directly treating the entire point cloud data as the root fragment P. (0) Then, consider fragment segmentation at subsequent levels h (starting from h=1). Using the parameters of level h, segment each fragment in hundreds of millions of levels. Divide into sub-fragments. This will produce a fragment candidate set S. (h) However, not all S (h)Both can be accepted. It is also needed to determine whether this deep level segmentation is beneficial by stopping the segmentation condition.

[0167] As an example, The semantic label of the father patch is Patch The semantic label of the father patch is The l i is assigned to the patch P i The score of the patch P i is defined as the sum of the sub-scores in the global score s({l i}), see equation 15, which is valid for any segmentation level. The determination of the semantic label of the patch P is explained in the following steps.

[0168]

[0169] Then, a normalized confidence score C

[0170]

[0171] The sufficient and necessary condition for accepting the sub-patch P is:

[0172]

[0173]

[0174] where equation 17 means that the sub-patch has a different semantic label than its father patch. Otherwise, no further segmentation of the patch at higher level should be performed in order to avoid over-segmentation. Equation 18 states that the confidence score of the sub-patch should be higher than the confidence score of the father patch.

[0175] It is understood that the stopping segmentation condition comprises two parts, one part is that the semantic label of the segmented patch is different from the semantic label of the patch before segmentation, and the other part is that the confidence score of the segmented patch is greater than the confidence score of the patch before segmentation, the confidence score of the patch is a parameter calculated according to equation 16. In the case of meeting the above two parts at the same time, the stopping segmentation condition is met.

[0176] S702, the segmented patch does not meet the stopping segmentation condition, and the segmented patch is segmented again in the next level.

[0177] The segmented patch does not meet the stopping segmentation condition, as an example, the segmented patch does not meet equation 17 and / or equation 18, which indicates that the segmented patch needs to be segmented again.

[0178] As an example, in the case that the segmented patch does not satisfy the stop segmentation condition, segmentation is continued until the segmented patch satisfies the stop segmentation condition.

[0179] S703, the segmented patch satisfies the stop segmentation condition, and the patch satisfying the stop segmentation condition is taken as a patch of the point cloud data.

[0180] In the case that the segmented patch satisfies the stop segmentation condition, i.e., both formula 17 and formula 18 are satisfied, segmentation is stopped, and the patch satisfying the stop segmentation condition is taken as a patch of the point cloud data.

[0181] In the case that the segmented patch satisfies the stop segmentation condition, i.e., both formula 17 and formula 18 are satisfied, segmentation is stopped, and the patch satisfying the stop segmentation condition is taken as a patch of the point cloud data. Figure 7 In the embodiment of the application, on the one hand, the point cloud data is segmented into multiple levels; on the other hand, the stop segmentation condition is used to determine whether the segmentation needs to be performed again. Thus, the point cloud data is reasonably segmented.

[0182] In the embodiment of the application, Level h The patch of the Level is composed of patches from the following three sources:

[0183] 1) The patch of the Level (h) is accepted as a sub-patch in the Level

[0184] 2) The patch of the Level is rejected as a sub-patch in the Level

[0185] 3) The patch of the Level

[0186] The result of the last level, i.e., the patch and the assigned semantic label, is regarded as the result of the multi-scale semantic segmentation.

[0187] That is, in the embodiment of the application, the multiple patches segmented from the point cloud data are segmented according to the levels. In the case that the patch satisfies the stop segmentation condition, the segmentation needs to be performed again. In this way, adaptive segmentation is implemented for different patches, and the segmentation level of each patch is determined by the stop segmentation condition. Then, the patch segmentation based on the semantic label is implemented.

[0188] In the embodiment of the application, the semantic label of the patch is described in detail as follows. Figure 7

[0189] Referring to Figure 8 , Figure 8 is a flowchart for assigning a semantic label to a patch according to the embodiment of the application, and specifically includes the following steps:

[0190] ​S801, calculate semantic confidence of a patch of point cloud data.

[0191] In embodiments of the present application, the multi-scale control algorithm, i.e. Figure 7 is used to implement the segmentation of patches, and single-scale semantic segmentation is used to assign semantic labels to patch headers. Single-scale semantic segmentation is the basis of the multi-scale control algorithm.

[0192] In point cloud data, there is no prior instance segmentation boundary, so it is impossible to limit the patches to have consistent semantic labels during the region growing process. Figure 8 The goal of the scheme in i is to assign a semantic label l i to each patch P base generated after the segmentation of the point cloud. i To this end, a total score s of semantic segmentation is defined, and the goal is to find the maximum value of s, and then the semantic label corresponding to the maximum value is taken as the semantic label of the patch.

[0193] The confidence conf base (P i ,c i ,l i ) is weighted and averaged to describe the relationship between the cluster c and the semantic label l i . The semantic label confidence weight of the patch is inversely proportional to the feature distance between the patch P i and the cluster center:

[0194]

[0195] wherein the weight w(P i ,c) is defined as:

[0196]

[0197] wherein d is the distance between the feature vector of the patch P i and the cluster center c. The distance definition can also be replaced by Mahalanobis Distance. However, in embodiments of the present application, the definition of d is more consistent with the cluster feature distance.

[0198] S802, take the semantic label corresponding to the maximum value of the semantic confidence as the semantic label of the patch of point cloud data.

[0199] For a patch, the semantic confidence of each semantic label is calculated according to formula 19. The semantic label corresponding to the maximum value of the semantic confidence is taken as the semantic label of the patch.

[0200] To determine the contextual rule confidence of the patch pair (P x ,P y ), embodiments of the present application define conf pc (Px y x y ) are measured as follows:

[0201]

[0202] where c x and c y are defined as the cluster centers closest to the patches P x and P y , q = (c x , c y , r xy , l x , l y ) is the corresponding five-tuple. In this way, the semantic label assignment problem can be achieved by minimizing the following energy:

[0203]

[0204] E x,y (l x , l y ) = 1 - conf pc (P x , P y , l x , l y ) Formula 25

[0205] where E({l i}) is the unary energy term, representing the "semantic-cluster" relationship. E x,y (l x , l x ) is the binary energy term, representing the "semantic-context" relationship. The energy definition of Formula 25 corresponds to Formula 23, and the goal is to find its minimum. However, the binary energy term does not necessarily satisfy the following property in general:

[0206] E x,y (l x , l x ) + E x,y (l y , l y ) ≤ E x,y (l x , l y ) + E x,y (l y , l x ) Formula 26

[0207] Satisfying Formula 26 is necessary for the Multi-label Graph-cut. It can be understood that the second term in Formula 19 can be calculated using Formulas 22-26.

[0208] However, in the embodiments of the present application, the confidence of some patch pairs assigned with different semantic combinations can be higher than the confidence of the same semantic. For example, the "keyboard-display" context patch cloud combination tends to indicate that they have different semantic labels. For the above case, the patch includes multiple semantic labels.

[0209] S102, merge the patches with the same semantic label into the same object instance according to the patch merging rule and the repulsive patch merging rule, and the patch merging condition and the repulsive patch merging condition are extracted from the cluster group of the training data.

[0210] The instance segmentation aims to segment the input point cloud into objects with independent semantic meaning. And each object can contain multiple geometric patches.

[0211] Referring to Figure 9 , Figure 9 is a flowchart of merging patches into the same object instance according to the embodiments of the present application, which specifically includes the following steps:

[0212] S901, construct a patch pair with two patches with the same semantic label.

[0213] In the embodiments of the present application, the semantic label is used to assist the implementation of instance segmentation of the scene. Specifically, according to the "cluster-instance" relationship, the patches with the same semantic label are merged into the same object instance according to the patch merging rule and the repulsive patch merging rule.

[0214] Therefore, in order to merge the patches, a patch pair is constructed with two patches of the point cloud data with the same semantic label.

[0215] Similar to the patch context of the semantic label, for the training set (P x ,P y )∈P, a patch "cluster-instance" relationship five tuple can be analogously obtained:

[0216] q′=(c x ,c y ,r′ xy ,l,pred) Formula 27

[0217] Wherein, c x and c y are the cluster group numbers corresponding to the patches P x and P y . r′ xy describes the patch P x and P yThe relationship is similar to the previously defined fragment relationship. However, the "adjacent" relationship is removed in this step because, for instance segmentation, merged fragments only consider the "closest" relationship. l refers to the semantic label of the fragment pair. Here, P is restricted... x and P y Fragments with the same semantic label have different semantic labels because fragments with different semantic labels obviously belong to different object instances. In the parallel comparison process without considering semantic labels, the semantic label l in all quintuples of Equation 27 is fixed at 0, indicating whether two fragments belong to the same object instance, defined as follows:

[0218]

[0219] Among them, o x and o y P represents the fragment. x and P y The instance number of the object to which it belongs.

[0220] Calculate the frequency of each quintuple, denoted as γ′. q′ In the statistical training set, clusters belonging to the same group and having the same geometric relation (r) as q′ are considered. x ′ y The number of fragment pairs of semantic tags can be used to further calculate the proportion τ′ that conforms to the same object instance relationship. q′ Defined as:

[0221]

[0222] In processing point cloud data, each fragment is first treated as a separate instance object. Assume A... x It belongs to o x The collection of fragments of the object instance o x Includes fragment P x Initially, we have:

[0223]

[0224] S902. If the fragment pair meets the proportion of the same object instance in the point cloud data, meets the fragment merging rule, and does not meet the exclusion fragment merging rule, then the fragments of the fragment pair will be merged into the same object instance.

[0225] To improve the speed of fragment processing, τ′ q′ If a larger value corresponds to a larger number of fragments, then τ′ q′ The fragment pairs are processed in descending order of iteration. Then, for each fragment pair, the arbitrary fragment pairs of its current instance are computed. The constructed "fragment-instance" relationship is defined as τ xy Fragment set A xand A y The sufficient and necessary conditions for merging into the same instance object are as follows:

[0226] l x = l y Formula 31

[0227]

[0228]

[0229] Formula 31 shows that the same semantic label is the premise of the merging of the fragments. For the set of instance fragments to be merged, there must be a merging relationship and there must not be any exclusion relationship. That is, the proportion of the fragment pair that represents the same object instance meets the fragment merging rule and does not meet the exclusion fragment merging rule.

[0230] Specifically, the semantic labels of the fragment pair are the same, the proportion of the fragment pair that represents the same object instance meets the fragment merging rule and does not meet the exclusion fragment merging rule.

[0231] Specifically, the Union-Find Data Structure data structure is used in the embodiment of the application to maintain the fragment merging process. The Union-Find Data Structure supports the merging and searching of sets in an average time complexity of O(1), which is very effective for optimizing the instance segmentation process. The process is repeatedly executed until all the mergable fragments are processed.

[0232] In Figure 9 In the embodiment, the fragments are merged into the same object instance in the form of a fragment pair, thereby realizing the segmentation of the point cloud data.

[0233] S103, segment the point cloud data according to the object instance.

[0234] After the object instance is determined, the point cloud data can be segmented according to the object instance. As an example, two object instances are determined, and then the point cloud data can be segmented into point cloud data corresponding to the two object instances.

[0235] In the above embodiment, the fragment merging condition and the exclusion fragment merging condition are used to merge the fragments into the object instance in the form of a fragment pair, thereby segmenting the point cloud data into multiple object instances. The under-segmentation and over-segmentation are avoided, and the time for segmenting the point cloud data can be shortened.

[0236] Specifically, shape features and context information of training point cloud data are utilized to perform semantic segmentation and instance segmentation on complex three-dimensional indoor scenes. Unsupervised patch shape clustering is combined with supervised rule learning method, and reliable context rules are extracted by using patch spatial relationship. Patch clustering groups serve as intermediate tools for context analysis, allowing parts of each object to belong to different clustering groups, and allowing the same clustering group to contain patches with different semantic labels. For example, patch segmentation of a single table is usually quite different in shape, and it is difficult to classify the table top and table leg as the same class in terms of geometry. However, the patch segmentation results of multiple tables have certain shape similarity, because the table top and table leg of different tables have a high probability of having similar geometric shapes.

[0237] Adaptive parameter scale selection is performed by learned context information. Multi-scale patch segmentation can greatly avoid the over-segmentation and under-segmentation problems caused by the region growing algorithm. The semantic labels of the patches are used to guide patch merging, and finally the point cloud is segmented into individual object instances.

[0238] Compared with traditional point cloud segmentation, the embodiment of the application realizes semantic segmentation and instance segmentation, and greatly avoids the phenomenon of patch over-segmentation and under-segmentation. Compared with existing geometric segmentation and recognition algorithms, the learning rate is high, the algorithm runs efficiently, and the unstructured point cloud data can be directly processed.

[0239] Reference Figure 10 , Figure 10 is a schematic diagram of the main structure of the point cloud data segmentation device according to the embodiment of the application, which can implement the point cloud data segmentation method as shown in Figure 10 The point cloud data segmentation device specifically includes:

[0240] The label module 1001 divides the point cloud data into patches of point cloud data, and assigns semantic labels to the patches of point cloud data;

[0241] The merging module 1002 is configured to merge the patches with the same semantic labels into the same object instance according to patch merging conditions and exclusion patch merging conditions, wherein the patch merging conditions and the exclusion patch merging conditions are extracted from the clustering groups of the training data;

[0242] The segmentation module 1003 is configured to segment the point cloud data according to the object instance.

[0243] In an embodiment of the application, the label module 1001 is further configured to divide the training data into training patches according to training semantic labels and training instance labels, wherein the training semantic labels are used to identify statistical features of object instances, and the training instance labels are used to identify geometric features of object instances.

[0244] The training patches are divided into the patch cluster groups by using a clustering algorithm to extract the patch merging condition and the repulsive patch merging condition in each patch cluster group.

[0245] In an embodiment of the present application, the label module 1001 is specifically configured to determine the spatial relationship of two training patches forming a patch pair according to the oriented bounding box of the training patches;

[0246] According to the spatial relationship of the patch pair, the cluster group of the patch pair and the training semantic label of the patch pair, the appearance frequency of the patch pair and the appearance proportion of the patch pair are extracted;

[0247] According to the appearance frequency and the appearance proportion, the patch merging condition and the repulsive patch merging condition are determined.

[0248] In an embodiment of the present application, the label module 1001 is specifically configured to split the point cloud data into patches of point cloud data;

[0249] The semantic confidence of the patches of point cloud data is calculated;

[0250] The maximum value of the semantic confidence corresponds to a semantic label, which is taken as the semantic label of the patch of point cloud data.

[0251] In an embodiment of the present application, the label module 1001 is specifically configured to divide the point cloud data into multiple levels, and split the point cloud data into multiple patches according to the current level;

[0252] If the split patch does not meet the stop splitting condition, the split patch is segmented again in the next level;

[0253] If the split patch meets the stop splitting condition, the patch meeting the stop splitting condition is taken as the patch of point cloud data.

[0254] In an embodiment of the present application, the stop splitting condition includes:

[0255] The semantic label of the split patch is different from the semantic label of the patch before splitting, and the confidence of the split patch is greater than the confidence of the patch before splitting.

[0256] In an embodiment of the present application, the merging module 1002 is specifically configured to construct a patch pair with two patches with the same semantic label;

[0257] If the patch pair meets the proportion of the same object instance in the point cloud data, meets the patch merging rule and does not meet the repulsive patch merging rule, the patches of the patch pair are merged into the same object instance.

[0258] Figure 11 An exemplary system architecture 1100 is shown, which can apply the method for segmenting point cloud data or the apparatus for segmenting point cloud data.

[0259] As shown in the Figure 11 , the system architecture 1100 can include terminal devices 1101, 1102, 1103, a network 1104 and a server 1105. The network 1104 is a medium for providing a communication link between the terminal devices 1101, 1102, 1103 and the server 1105. The network 1104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0260] The user can use the terminal devices 1101, 1102, 1103 to interact with the server 1105 through the network 1104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 1101, 1102, 1103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0261] The terminal devices 1101, 1102, 1103 can be various electronic devices with display screens and support for web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0262] The server 1105 can be a server that provides various services, such as a background management server that provides support for shopping websites browsed by users using the terminal devices 1101, 1102, 1103 (only as an example). The background management server can analyze and process received product information query requests and other data, and feed back the processing results (such as target push information, product information - only as examples) to the terminal devices.

[0263] It should be noted that the method for segmenting point cloud data provided by the embodiments of the present application is generally executed by the server 1105, and correspondingly, the apparatus for segmenting point cloud data is generally provided in the server 1105.

[0264] It should be understood that Figure 11 the number of terminal devices, networks and servers in the above description is only illustrative. Any number of terminal devices, networks and servers can be provided according to the needs of implementation.

[0265] Reference is made to Figure 12 , which shows a structural diagram of a computer system 1200 of a terminal device suitable for implementing the embodiments of the present application. Figure 12The terminal device shown is merely an example and should not impose any limitation on the functions and usage range of the embodiments of the present application.

[0266] As Figure 12 shown, the computer system 1200 includes a central processing unit (CPU) 1201 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 1202 or a program loaded from the storage section 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the system 1200 are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0267] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable media 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read out therefrom is installed into the storage section 1208 as necessary.

[0268] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with the embodiments disclosed herein. For example, the embodiments disclosed herein include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1209 and / or installed from the removable media 1211. When the computer program is executed by the central processing unit (CPU) 1201, the above-described functions defined in the system of the present application are performed.

[0269] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0270] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0271] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The modules described can also be arranged in a processor, for example, a processor can be described as including a label module, a merging module and a segmentation module. In some cases, the names of these modules do not constitute a limitation on the modules themselves, for example, the label module can also be described as "segmenting point cloud data into point cloud data patches and assigning semantic labels to the point cloud data patches".

[0272] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0273] segmenting point cloud data into point cloud data patches and assigning semantic labels to the point cloud data patches;

[0274] merging the patches with the same semantic labels into the same object instance according to patch merging conditions and repulsive patch merging conditions, the patch merging conditions and the repulsive patch merging conditions being extracted from the cluster group of the training data;

[0275] segmenting the point cloud data according to the object instance.

[0276] According to the technical solutions of the embodiments of the present application, point cloud data is segmented into point cloud data patches, and semantic labels are assigned to the point cloud data patches; the patches with the same semantic labels are merged into the same object instance according to patch merging conditions and repulsive patch merging conditions, the patch merging conditions and the repulsive patch merging conditions being extracted from the cluster group of the training data; and the point cloud data is segmented according to the object instance. The patches belonging to the same object instance are merged by using the patch merging conditions and the repulsive patch merging conditions, and finally the point cloud data is segmented into multiple object instances, so as to avoid under-segmentation and over-segmentation, and the time for segmenting the point cloud data can be shortened.

[0277] The specific embodiments described above do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of segmenting point cloud data, the method comprising: The method comprises the following steps: segmenting the point cloud data into multiple levels, and cutting the point cloud data into multiple patches according to the current level; if the cut patches do not meet the stop cutting condition, the cut patches are segmented again in the next level; if the cut patches meet the stop cutting condition, the patches meeting the stop cutting condition are taken as the patches of the point cloud data, and a semantic label is assigned to the patches of the point cloud data; two patches with the same semantic label are taken to construct a patch pair; if the patch pair meets the proportion of the same object instance in the point cloud data, meets the patch merging rule, and does not meet the exclusion patch merging rule, the patches of the patch pair are merged into the same object instance; the patch merging condition and the exclusion patch merging condition are context rules extracted from the clustering group results of training data; the clustering group results are used as the intermediate representation of data, so that the context rules including the patch merging condition and the exclusion patch merging condition can be more reliably analyzed and extracted; the point cloud data is segmented according to the object instance.

2. The method of claim 1, wherein, The method further comprises the following steps: training data is segmented into training patches according to training semantic labels and training instance labels; the training semantic labels are used to identify statistical features of object instances; the training instance labels are used to identify geometric features of object instances; a clustering algorithm is used to divide the training patches into patch clustering groups, so as to extract the patch merging condition and the exclusion patch merging condition in each patch clustering group.

3. The method of segmenting point cloud data according to claim 2, wherein, The extraction of the patch merging condition and the exclusion patch merging condition in each patch clustering group comprises the following steps: the spatial relationship of two training patches constituting a patch pair is determined according to the oriented bounding box of the training patches; the occurrence frequency of the patch pair and the occurrence proportion of the patch pair are extracted according to the spatial relationship of the patch pair, the clustering group of the patch pair, and the training semantic label of the patch pair; the patch merging condition and the exclusion patch merging condition are determined according to the occurrence frequency and the occurrence proportion.

4. The method of claim 1, wherein, The cutting of the point cloud data into patches of the point cloud data and the assignment of semantic labels to the patches of the point cloud data comprise the following steps: the point cloud data is cut into patches of the point cloud data; the semantic confidence of the patches of the point cloud data is calculated; the maximum value of the semantic confidence corresponds to a semantic label, which is taken as the semantic label of the patches of the point cloud data.

5. The method of claim 1, wherein, The stop cutting condition comprises the following steps: the semantic label of the cut patches is different from the semantic label of the uncut patches, and the confidence of the cut patches is greater than the confidence of the uncut patches.

6. An apparatus for segmenting point cloud data, the apparatus comprising: The method comprises the following steps: a label module is configured to segment the point cloud data into multiple levels, and cut the point cloud data into multiple patches according to the current level; if the cut patches do not meet the stop cutting condition, the cut patches are segmented again in the next level; if the cut patches meet the stop cutting condition, the patches meeting the stop cutting condition are taken as the patches of the point cloud data, and a semantic label is assigned to the patches of the point cloud data. The merging module is configured to merge two of the patches with the same semantic label into a patch pair, and merge the patches of the patch pair into the same object instance if the patches of the patch pair meet the proportion of the same object instance in the point cloud data, meet the patch merging rule, and do not meet the exclusion patch merging rule; the patch merging condition and the exclusion patch merging condition are context rules extracted from clustering group results of training data. The segmentation module is configured to segment the point cloud data according to the object instance. 7.An electronic device for segmenting point cloud data, comprising: The method comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-5.

Images