Methods for obtaining object positions and point cloud segmentation

Abstract

A method for obtaining the position of an object and a method for point cloud segmentation, wherein the method for point cloud segmentation first constructs a feature vector of a point cloud using point cloud data, then obtains the initial probability distribution of the category of the point cloud, and finally inputs the initial probability distribution and feature vector of each point cloud category into a pre-trained probability graph model to obtain the probability distribution of the category of the point cloud, and takes the category of the point cloud with the highest probability in the probability distribution of each point cloud category. The random variable of the probability graph model is the category of the point cloud, and each random variable is independent of each other. Its energy function includes a unary part and a binary part. The initial probability distribution is used to initialize the unary part, and the binary part is a function representation of the feature vector pairs formed between the current point cloud and other point clouds. Therefore, when segmenting, not only the information of the current point cloud is considered, but also the relationship between the point clouds is considered, which is conducive to more complete segmentation of objects and backgrounds and improves the effect of point cloud segmentation.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method for obtaining an object position and a method for point cloud segmentation. Background Art

[0002] In the field of robotics, autonomous grasping of target objects is a key capability of intelligent robots. Among them, grasping of scattered objects has always been the key to realizing the intelligence of robots. However, it takes several weeks for current robots to reprogram to perform a complex new grasping task, which makes the reorganization of modern manufacturing production lines very expensive and slow. Moreover, robots are mostly used in specific environments to grasp specific known objects. For unknown objects placed in different postures in uncertain environments, it is not yet mature to let robots autonomously determine the grasping position of the grasped object and the grasping posture of the grasping gripper. If the robot can autonomously grasp scattered objects, it can reduce the time of robot teaching programming, better realize the flexibility and intelligence of automated manufacturing, adapt to the current multi-category and small-batch production needs, and meet the needs of rapid updating of manufacturing equipment when products are replaced.

[0003] Point cloud is an effective three-dimensional object representation tool with high flexibility and scalability. It is often used by robots to obtain and analyze information such as the position, posture, and contour of objects. Effective segmentation of point cloud is an important basis for robots to grasp objects, trajectory planning, and obstacle avoidance. Accurate point cloud segmentation and object category recognition are conducive to enabling robots to select appropriate tools and postures to complete tasks such as grasping and picking. Point clouds are mainly obtained through three-dimensional scanners, or through two-dimensional cameras to obtain two-dimensional images and then perform three-dimensional reconstruction. The purpose of point cloud segmentation is to distinguish different objects in the point cloud through ideas similar to semantic segmentation or instance segmentation, so as to analyze information such as position, posture, and contour based on object information. The current analysis and segmentation of point clouds are mainly based on the shape information of objects in the scene, such as planes, cylinders, spheres, etc. These geometric shapes can be expressed and described in geometric parameterized forms, so these objects can be segmented and processed more stably.

[0004] Currently, there are two main types of point cloud segmentation methods: one is a point cloud segmentation method based on traditional methods, and the other is a point cloud segmentation method based on deep learning.

[0005] Traditional methods of point cloud segmentation include edge-based segmentation, region-growing-based segmentation, and model-based segmentation. These methods first convert the unordered data of the point cloud into a KD tree or octree structure, establish the topological relationship of the point cloud, convert the unordered data into ordered data, then obtain the features of the point cloud based on the topological relationship, and finally divide and segment according to the corresponding feature criteria designed. Edge segmentation is to judge the edge based on the drastic change of the point cloud normal vector and use it as the segmentation criterion. Region-growing-based segmentation is to select seed points in the point cloud and iteratively search for points with small curvature changes nearby to form a point cloud. Model-based segmentation requires matching the known model with the scene point cloud, and taking the matched point cloud near the model as the output result. Edge-based segmentation methods and region-growing-based segmentation methods cannot solve the segmentation problem when the point cloud is connected to the background in small blocks, and may cause over-segmentation, dividing an object into several pieces; while model-based segmentation methods require the prior knowledge of the three-dimensional model of the object to be segmented, and the application scenarios are limited. These methods all need to establish topological relationships between point clouds based on disordered point clouds, which is time-consuming to calculate. In the process of robot grasping, depalletizing, and stacking, the point cloud needs to be segmented quickly. The traditional method of establishing point cloud topological relationships is time-consuming, and due to the influence of factors such as point cloud noise and object shape, the segmented boundaries may appear inside the object or between the object and the background, and the boundaries are not completely continuous, so the segmented objects may not be complete.

[0006] The deep learning-based methods are currently mainly based on network structures such as MLP (Multi Layer Perceptron) and CNN (Convolutional Neural Networks), such as PointNet, PointNet++, etc. The network parameters are obtained by training with a data set with labeled information, and then reasoning is performed based on the network parameters to achieve point cloud segmentation. The segmentation method based on deep learning extracts features from point cloud data, and uses deep learning / machine learning models to learn object types of different categories, and then uses the learned model to classify the acquired point cloud data to complete point cloud segmentation. Deep learning algorithms based on convolutional neural networks have brought great improvements to the accuracy of point cloud segmentation, but they have high requirements for GPU (Graphics Processing Unit), and the training model requires a large amount of data. When the point cloud scene is too large, the point cloud needs to be regionalized and sampled first, and then segmented on the sampled point cloud data, which will also result in poor boundary effects of the segmented objects. In summary, the existing technology has the following deficiencies:

[0007] (1) Edge-based segmentation methods and region growing-based segmentation methods cannot solve the segmentation problem when there are small blocks connected between the point cloud and the background. Over-segmentation may occur, dividing an object into several pieces.

[0008] (2) Model-based segmentation methods require prior knowledge of the three-dimensional model of the object to be segmented, which limits their application scenarios.

[0009] (3) Establishing the topological relationship between point clouds based on unordered point clouds is computationally time-consuming.

[0010] (4) The deep learning-based segmentation method requires a large amount of data for annotation and training, has high requirements on hardware, and is difficult to process scenes with too much point cloud data, which can easily lead to poor segmentation results. Summary of the invention

[0011] The present invention provides a method for obtaining the position of an object and a method for point cloud segmentation, in the hope of solving the problem that the existing point cloud segmentation technology is difficult to completely segment the object and the background.

[0012] According to the first aspect, an embodiment provides a method for obtaining a position of an object, comprising:

[0013] Acquire point cloud data of the object, wherein the point cloud data includes position information of the point cloud;

[0014] constructing a feature vector of a point cloud using the point cloud data;

[0015] Initialize the probability that each point cloud belongs to each category and obtain the initial probability distribution p of the point cloud category i =[p i1 , p i2 ,…,p ic ], where i represents the i-th point cloud, p ij It represents the probability that the i-th point cloud belongs to the j-th category. The number of categories is c. The category of the point cloud represents the object to which the point cloud belongs.

[0016] For each point cloud, the initial probability distribution p of the point cloud category is i The feature vector of the point cloud is input into a pre-trained probability graph model to obtain the probability distribution of the category of the point cloud, wherein the random variables in the probability graph model are the categories of the point cloud, and each random variable is independent of each other, the energy function of the probability graph model includes a unary part and a binary part, and the initial probability distribution p i Used to initialize the unary part, the binary part is a function representation of the feature vector pairs formed between the current point cloud and other point clouds;

[0017] Take the point cloud category with the highest probability in the probability distribution as the category of this point cloud;

[0018] According to the preset transformation matrix, the point cloud and its categories are mapped to an image, the image is subjected to connectivity analysis to obtain connected regions, and the connected regions are screened according to preset conditions to obtain the region where each object is located;

[0019] The center of gravity of the area where each object is located is obtained, and the area where the object is located and its center of gravity are mapped to the point cloud according to the transformation matrix to obtain the minimum circumscribed rectangle and center of gravity of the object in space.

[0020] According to the second aspect, an embodiment provides a method for point cloud segmentation, including:

[0021] Acquire point cloud data, wherein the point cloud data includes location information of the point cloud;

[0022] constructing a feature vector of a point cloud using the point cloud data;

[0023] Initialize the probability that each point cloud belongs to each category and obtain the initial probability distribution p of the point cloud category i =[p i1 , p i2 ,…,p ic ], where i represents the i-th point cloud, p ij represents the probability that the i-th point cloud belongs to the j-th category, and the number of categories is c;

[0024] For each point cloud, the initial probability distribution p of the point cloud category is i The feature vector of the point cloud is input into a pre-trained probability graph model to obtain the probability distribution of the category of the point cloud, wherein the random variables in the probability graph model are the categories of the point cloud, and each random variable is independent of each other, the energy function of the probability graph model includes a unary part and a binary part, and the initial probability distribution p i Used to initialize the unary part, the binary part is a function representation of the feature vector pairs formed between the current point cloud and other point clouds;

[0025] The point cloud segmentation is completed by taking the point with the highest probability in the probability distribution of each point cloud category as the category of this point cloud.

[0026] In one embodiment, the probability graph model is expressed by the formula:

[0027]

[0028] in x i is a random variable representing the category of the i-th point cloud, Q i (x i) is the probability distribution of the category of the i-th point cloud, l represents a certain category, l′ represents all other categories except l, μ(l, l′) is the label consistency term, w m is the weight, f i is the feature vector of the i-th point cloud, h (m) (f i , f j ) is the mth binary term, used to characterize the relationship between two point clouds, K is the number of binary terms, is a unitary part and is the binary part, Z i is the normalization factor and

[0029]

[0030] In one embodiment, the probability graph model calculates the probability distribution of the point cloud category in the following way:

[0031] First, Then iterate the following steps until Q i (x i )convergence:

[0032] For all m, calculate

[0033] calculate

[0034] make

[0035] Q i (x i ) are indexed and normalized.

[0036] In one embodiment, the feature vector of the point cloud includes the position information and the normal vector of the point cloud, K=2, and

[0037]

[0038] Among them, p i Represents the location information of the i-th point cloud, n i represents the normal vector of the i-th point cloud, σ 1 , σ 2 and σ 3 is the preset parameter, the eigenvector f i =(p i , n i ) is a six-dimensional vector.

[0039] In one embodiment, the method for obtaining the position of an object and the method for segmenting a point cloud further include: iPerform indexation and normalization;

[0040] Among them, p i or Q i (x i ) is indexed and normalized as follows:

[0041] Get the maximum value in a probability distribution Calculate according to the following formula To replace the original probability value ζ ij :

[0042]

[0043] where ζ i =[ζ i1 , i2 ,…,ζ ic ] indicates p i or Q i (x i )’s probability distribution.

[0044] In one embodiment, the probabilistic graphical model is trained in the following manner:

[0045] Get point cloud sample data and corresponding category labels;

[0046] Construct the feature vector of the point cloud using the point cloud sample data;

[0047] Initialize the probability that each point cloud belongs to each category and obtain the initial probability distribution p′ of the point cloud category i ;

[0048] The initial probability distribution p′ of the point cloud category i The feature vector of the point cloud is input into the probability graph model, and the weight w is updated using the gradient descent method according to the obtained results and category labels. m and the label consistency term μ(x i , x j ), where the label consistency term μ(x i , x j ) is specifically updated iteratively according to the following formula:

[0049]

[0050] in, The superscript n represents the nth iteration, λ is the preset learning rate, is the category label item of the i-th point cloud, when x i If the value is the same as the category label, it is 1, otherwise it is 0.

[0051] In one embodiment, the probability of each point cloud belonging to each category is initialized to obtain the initial probability distribution p′ of the point cloud category. i ,include:

[0052] Set the initial probability of the same category as the class label to p e = -logP GT , the probability of other categories is set to p n = -log[(1-P GT ) / (c-1)], thereby obtaining the initial probability distribution p′ of the point cloud category i , where P GT is the preset confidence standard value.

[0053] In one embodiment, the probability of each point cloud belonging to each category is initialized to obtain the initial probability distribution p of the point cloud category. i ,include:

[0054] The feature vector of each point cloud is input into the trained machine learning classification model to obtain the probability that each point cloud belongs to each category, which is used as the initial probability distribution p of the point cloud category. i .

[0055] According to the third aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement the method described in the first aspect or the second aspect.

[0056] According to the method for obtaining the position of an object and the method for point cloud segmentation in the above embodiment, the feature vector of the point cloud is first constructed using the point cloud data, and then the probability of each point cloud belonging to each category is initialized to obtain the initial probability distribution of the point cloud category. Finally, the initial probability distribution p of each point cloud category is converted into the initial probability distribution p of each point cloud category. i The feature vector of the point cloud is input into the pre-trained probability graph model to obtain the probability distribution of the point cloud category, and the category with the highest probability in the probability distribution of each point cloud category is taken as the category of this point cloud, thereby completing the point cloud segmentation. Among them, the random variables in the probability graph model are the categories of the point cloud, and each random variable is independent of each other. Its energy function includes a unary part and a binary part. The initial probability distribution p iThe unary part is used to initialize the energy function, while the binary part is a function representation of the feature vector pairs formed between the current point cloud and other point clouds. Since the binary part in the probability graph model is a function representation of the feature vector pairs formed between the current point cloud and other point clouds, not only the information of the current point cloud is considered during segmentation, but also the relationship between the point clouds is considered, which is conducive to a more complete segmentation of the object and the background, and improves the effect of point cloud segmentation. The probability graph model of the present invention is obtained based on the independence of each random variable, so that each point cloud can independently calculate the probability distribution of the category, which is conducive to reducing the amount of calculation, improving the calculation speed, and reducing the requirements for hardware. Compared with the model-based segmentation method, the method of the present invention does not need to know the three-dimensional model of the object to be segmented in advance, and has a wider application scenario. The position of the object is obtained by the point cloud segmentation method of the present invention, and the position of the object can be obtained more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a method for point cloud segmentation according to an embodiment;

[0058] Figure 2 A flowchart of training a probabilistic graphical model in one embodiment;

[0059] Figure 3 The present invention is a flowchart of a method for obtaining a position of an object according to an embodiment. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0061] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0062] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0063] As described in the background technology, point cloud segmentation is of great significance for robot grasping objects. In an unstructured or semi-structured environment, the three-dimensional scanning equipment is used in conjunction with the robot to complete the grasping tasks such as depalletizing and stacking. It is necessary to complete the segmentation of the point cloud to obtain the posture and position of the object through the surface information of the object after segmentation. At present, the design and production of commodities are heterogeneous, which greatly increases the difficulty of three-dimensional morphological analysis of products, especially some complex special-shaped surfaces. Complex special-shaped surfaces have the characteristics of unstructured shape, arbitrary stacking in space, large curvature changes and difficult edge detection. Complex special-shaped surfaces are more and more widely used in high-end manufacturing fields such as 3C electronic manufacturing, automobile manufacturing, and smart wear. The demand for how to obtain three-dimensional morphological information of complex special-shaped surfaces and perform accurate analysis in industry is growing. Due to the high reflection and occlusion of complex special-shaped surfaces, the point cloud of complex special-shaped surfaces obtained by the three-dimensional scanning equipment is unstructured and has a lot of noise. As an important pre-processing process for measuring, grasping, and sorting three-dimensional objects, the effect of point cloud segmentation directly affects the accuracy of subsequent operations.

[0064] In the point cloud segmentation of complex and irregular surfaces, the current point cloud segmentation algorithms mainly have the following difficulties and problems:

[0065] (1) Complex irregular surface structures have the characteristics of unstructured shapes, compound curvature of surfaces, and arbitrary stacking in space. Therefore, it is difficult to find feature points and fit them, which makes segmentation difficult.

[0066] (2) Due to the large curvature variation of complex and irregular surfaces and the difficulty in edge detection, background points and edge noise points are easily mis-segmented into the foreground, making it impossible to achieve the accuracy required for precise measurement;

[0067] (3) The data volume of complex and irregular surface point clouds and their background point clouds is large, and the processing process is slow, which is not suitable for industrial manufacturing fields with high efficiency requirements.

[0068] In order to solve some problems existing in the prior art and completely segment the object and the background, the present invention provides a point cloud segmentation method, which integrates the traditional point cloud processing idea and the deep learning processing idea, and converts it into a probability-based statistical analysis idea for processing. It not only considers the current point cloud information, but also considers the relationship between the current point cloud and other point clouds, forming a binary probability statistical analysis result, which includes a univariate part and a binary part. The final probability distribution of the point cloud category is obtained from these two parts, and judgment is made based on this.

[0069] In the method of the present invention, the category of the point cloud is taken as a random variable, and the probability distribution of the category of each point cloud is obtained by means of a probabilistic graphical model. Since the point cloud has irregular and disordered characteristics, an undirected graph can be constructed to represent the relationship between the point clouds. The undirected graph here is a kind of probabilistic graphical model, for example, it can be a conditional random field, etc. The node in the graph corresponds to a point cloud, and the edge corresponds to the relationship between the point clouds, which can be expressed as G(V,E), where V represents the node, representing the feature information of the point cloud, and E represents the edge. When two point clouds are connected, there is an edge connecting the two point clouds. Here, every two point clouds are regarded as having a connection relationship. Due to the huge amount of point cloud data, the calculation of this problem is very complicated, requiring a lot of time and space, and is a dense point cloud processing problem. Therefore, based on the assumption of independent distribution of random variables, the present invention transforms the original model and derives a simpler model to replace the original model, which reduces the complexity of the calculation and reduces the demand for time and space.

[0070] Please refer to Figure 1 , a point cloud segmentation method in one embodiment of the present invention includes steps 110 to 150, which are described in detail below.

[0071] Step 110: Obtain point cloud data. Point cloud data can be obtained by scanning the surface of an object with a 3D scanner, or by acquiring a 2D image of the surface of an object with a 2D camera and then performing 3D reconstruction. Point cloud data represents information about points on the surface of the object. Point cloud data includes 3D position information (x, y, z) of the point cloud, and may also include normal vectors, RGB color information, depth information, etc.

[0072] Step 120: Construct a feature vector of the point cloud using the point cloud data.

[0073] The feature vector can be constructed using one or more information in the point cloud data. In one embodiment, the feature vector can be constructed using the position information and normal vector of the point cloud to obtain a six-dimensional feature vector f i =(p i , n i ), where the subscript i represents the i-th point cloud, p i represents the location information of the i-th point cloud, ni Represents the normal vector of the i-th point cloud.

[0074] When the acquired point cloud data does not have a normal vector, normal vector estimation is required. The process of normal vector estimation is as follows:

[0075] (1) Construct a KD tree based on the point cloud data. Search the KD tree according to the preset search radius to obtain the K-nearest neighbor points of the point cloud. Use the K-nearest neighbor points to fit a plane. The plane passes through the center of gravity of the data set composed of the K-nearest neighbor points. Then the equation of the plane can be expressed as:

[0076]

[0077] Where (x i ,y i , z i ) are K-nearest neighbor points, is the center of gravity, a, b, c are the coefficients of the plane equation.

[0078] (2) Obtain the centroid of the data set:

[0079]

[0080] Where N is the total number of data in the dataset.

[0081] (3) Remove the center of gravity of the data set, that is,

[0082]

[0083] (4) According to the following optimization objectives, the coefficients a, b, and c are obtained using the least squares method:

[0084]

[0085] Thus, the normal vector (a, b, c) of the point cloud is obtained.

[0086] Before using point cloud data for point cloud segmentation, the point cloud data can be gridded first, and then the gridded point cloud data can be used for point cloud segmentation. The gridding process is as follows:

[0087] (1) Calculate the maximum and minimum values ​​of the x, y, and z coordinate components, which are x min 、x max ,y min , y max , z min 、z max , can be obtained

[0088] d x =x max -x min , d y =ymax -y min , d z =z max -z min ,

[0089] Based on this, the radius of the outer sphere of all point clouds can be calculated as: Set the sampling steps of x, y, and z to be δ x ,δ y and δ z , discretize the point cloud data into voxel grids. Since the point cloud coordinates (x i ,y i , z i ) is a unit vector, so the maximum possible numbers of x, y, and z after discretization are After quantizing the point cloud coordinates, the index values ​​are calculated. Point clouds with the same index values ​​are stored in the same array. The index values ​​are calculated as follows: First, calculate

[0090]

[0091] Then the index value index = x cell n y n z +y cell n z +z cell .

[0092] (2) To complete clustering calculation for point clouds with the same index value, mean clustering or Gaussian clustering can be used.

[0093] Mean clustering: Let c be the number of point clouds with the same index value for the i-th index value. i , the output after clustering is a total number of different point cloud coordinate-normal vector pairs. The point cloud coordinates are calculated according to the following formula:

[0094]

[0095] The normal vector is calculated according to the following formula:

[0096]

[0097] Gaussian clustering: First, calculate the centroid of all point clouds (x c ,y c , z c ), then calculate the distance from the point cloud to the center of gravity, calculate the weight based on the distance, and then calculate the weighted sum. The distance from the point cloud to the center of gravity is in

[0098] Δx=x i -x c , Δy=yi -y c , Δz=z i -z c ,

[0099] Weight or α i =1 / d i , all weights and Similarly, the output after clustering is different point cloud coordinate-normal vector pairs, and the point cloud coordinates are calculated according to the following formula:

[0100]

[0101] The normal vector is calculated according to the following formula:

[0102]

[0103] Step 130: Initialize the probability that each point cloud belongs to each category, and obtain the initial probability distribution p of the point cloud category i =[p i1 , p i2 ,…,p ic ], where i represents the i-th point cloud, p ij It represents the probability that the i-th point cloud belongs to the j-th category, and the number of categories is c.

[0104] In one embodiment, a machine learning method can be used to classify point clouds to obtain the initial probability distribution of point cloud categories. First, the feature vector and category label of the point cloud sample are used to train the machine learning classification model. When performing point cloud segmentation, the feature vector of each point cloud is input into the trained machine learning classification model to obtain the probability that each point cloud belongs to each category, which is used as the initial probability distribution p of the point cloud category. i The machine learning classification model can be a neural network or SVM (Support Vector Machine), etc.

[0105] After obtaining the initial probability distribution p i After that, it can also be indexed and normalized, specifically: Get the initial probability distribution p i The maximum value in Calculate according to the following formula To replace the original probability value p ij :

[0106] Step 140: For each point cloud, the initial probability distribution p of the category of the point cloud i And the feature vector is input into the pre-trained probability graph model to obtain the probability distribution of the category of the point cloud.

[0107] As mentioned above, an undirected graph can be used to model the point cloud. When designing the energy function or loss function in the undirected graph, not only the information of the current point cloud is considered, but also the relationship between the point clouds. Based on this consideration, the designed energy function includes two parts, one is a unary function related only to the current point cloud, and the other is a binary relational function that describes the relationship between the current point cloud and other point clouds. Since the relationship between point clouds is described by an undirected graph, the relationship matrix between point clouds is a symmetric matrix. We only need to consider the upper diagonal or lower diagonal of the matrix, so the energy function can be defined as:

[0108]

[0109] where x i is a random variable representing the category of the i-th point cloud. Here each point cloud has a unary function In this function, only the category of the current point cloud is considered, and the categories of other point clouds are not considered for the time being. It is a binary function. Each random variable of a point cloud category calculates an energy feature with the random variables of other point cloud categories to characterize the "closeness" between the two. In this energy function, the binary part needs to calculate the joint energy of all point clouds. Assuming there are n point clouds, there will be n(n-1) / 2 possible combinations of the binary part. In general, the number of energy features is the square of the number of point clouds. If we have 1 million point cloud data, we need to establish 495 billion sets of energy features, which requires a lot of calculation and storage.

[0110] The unary part can use the initial probability distribution p of the point cloud category i To assign values, the binary part can be calculated using the feature vector of the point cloud in the following form:

[0111]

[0112] where h (m) (f i , f j ) is the mth binary term representing the relationship between two point clouds, w m is its weight, and K is the number of binary terms. μ(x i , x j) is called the label consistency term, which constrains the conditions for the conduction of the connection relationship of the edges in the undirected graph. Only when the label categories are the same can the energy be mutually conducted. For example, the energy of "a point cloud may be a bird" can be mutually conducted with the energy of "another point cloud may be a bird", thereby increasing or decreasing the energy of the latter "may be a bird", thereby affecting the probability of the latter "may be a bird"; and the energy of "a point cloud may be a bird" cannot affect the probability of "another point cloud may be the sky", which is an item that is approximately non-zero or one. The weight here can also be designed with a soft threshold. The binary part represents the "closeness" between different point clouds in the form of features.

[0113] When the six-dimensional feature vector f is constructed using the position information and normal vector of the point cloud i =(p i , n i ), the binary part can be designed as:

[0114]

[0115] That is, K = 2 at this time, and

[0116] where σ 1 , σ 2 and σ 3 are preset parameters.

[0117] Here, we consider both the position and normal vector information for comprehensive judgment and analysis. It can be seen that if the distance between two point clouds is close and the normal vectors are close, then the energy feature between them is stronger and the "closeness" is higher, otherwise it is not high. At the same time, the parameter σ in the denominator 1 , σ 2 and σ 3 It also controls the strength of the energy feature. The larger the denominator, the weaker the energy feature. This is equivalent to finding similar point cloud pairs in a six-dimensional space and strengthening their energy features.

[0118] After designing the probability graph model, we can calculate the joint probability distribution of the point cloud categories, and take the result with the largest joint probability distribution as the category of the point cloud. The amount of calculation is very large, and solving the joint probability distribution of an undirected graph is still a task with a lot of calculation, so it is necessary to take measures to reduce the computational complexity of the model. The idea of ​​the present invention is to use a simple model to replace the original complex model, and the replaced model should be as close to the original model as possible. After completing the replacement of the model, the new model can be used for calculation, and the original complex model can no longer be used. The present invention derives a simpler new model based on the independence assumption.

[0119] Based on the energy function of formula (1), the joint probability distribution of undirected graphs can be expressed as:

[0120]

[0121] Where Z is the normalization factor, which is the sum of the potential functions of all possible category distributions, that is,

[0122]

[0123] For the original distribution P(X), let the distribution of the new model be Q(X), where the random variables are independently distributed, then:

[0124]

[0125] Where Q i (x i ) is the probability distribution of the category of the i-th point cloud.

[0126] For P(X) and Q(X), we need a suitable way to evaluate the closeness between them. We can use KL divergence as the loss function to evaluate and derive a new model based on it. Let the loss function of KL divergence be expressed as:

[0127]

[0128] Since the sum of the probabilities of each category of each point cloud is 1, constraints can be added: Substituting equation (2) and equation (3) into the above equation, we can obtain the following equation according to the Lagrange multiplier method:

[0129]

[0130] Further simplifying the above formula, we can get

[0131]

[0132] in l represents a certain category, l′ represents all other categories except l, and Z i is the normalization factor and

[0133]

[0134] Formula (4) is the new model obtained. It can be seen that it also includes a univariate part and a binary part. is a unitary part and The binary part.

[0135] In this way, the new model can be used to calculate the probability distribution of each point cloud category, without the need to use the original more complex model, which greatly improves the calculation speed. The calculation process using the above new model is as follows:

[0136] First, initialize it, Then iterate the following steps until Q i (x i )convergence:

[0137] For all m, calculate

[0138] calculate

[0139] make

[0140] Q i (x i ) is subjected to exponential and normalization processing. The exponential and normalization processing here is the same as the initial probability distribution p i The exponentiation and normalization treatments are the same.

[0141] When the number of iterations reaches the preset value or the change of the results of two iterations is less than the preset value, it is considered to have converged. The change of the iteration result can be calculated (Q obtained in this iteration i (x i )-Q obtained in the previous iteration i (x i )) / Q obtained in the last iteration i (x i ) to indicate that when the calculated result is less than the preset threshold, Q is determined i (x i ) has converged, thus obtaining a stable probability distribution Q of the point cloud category i (x i ).

[0142] Step 150: Take the point cloud category with the highest probability in the probability distribution as the category of the point cloud, thereby completing the point cloud segmentation.

[0143] Before using the above new model for calculation, it is necessary to train it to obtain the weight w m , label the consistency term μ(x i , x j ) and other parameters, please refer to Figure 2 ,The model training process includes steps 161 to 164 which are described below.

[0144] Step 161: Obtain point cloud sample data and corresponding category labels.

[0145] Step 162: construct a feature vector of the point cloud using the point cloud sample data. The construction of the feature vector can refer to step 120. When the acquired point cloud sample data does not include a normal vector, normal vector estimation can also be performed, and the point cloud sample data can also be gridded.

[0146] Step 163: Initialize the probability that each point cloud belongs to each category, and obtain the initial probability distribution p′ of the point cloud category i .

[0147] In one embodiment, the probability of each point cloud belonging to each category can be initialized by setting the initial probability of the category with the same category label to p e = -logP GT , the probability of other categories is set to p n = -log[(1-P GT ) / (c-1)], thereby obtaining the initial probability distribution p′ of the point cloud category i , where P GT is a preset confidence standard value, which can be set to 0.9, for example.

[0148] Step 164: Initial probability distribution p′ of the point cloud category i The feature vector of the point cloud is input into the new model to obtain the category of each point cloud. According to the obtained results and category labels, the weight w is updated using the gradient descent method. m and the label consistency term μ(x i , x j ). where the label consistency term μ(x i , x j )The specific update formula is:

[0149]

[0150] in, The superscript n represents the nth iteration, λ is the preset learning rate, is the category label item of the i-th point cloud, when x i If the value is the same as the category label, it is 1, otherwise it is 0.

[0151] After completing the model training through the above process, you can use the trained new model for point cloud segmentation.

[0152] According to the method for point cloud segmentation of the above embodiment, the probability distribution of the point cloud category is obtained using the probability graph model, and then the category with the largest probability is taken as the category of the point cloud, thereby completing the point cloud segmentation. Among them, the random variables in the probability graph model are the categories of the point cloud, and each random variable is independent of each other. Its energy function includes a one-dimensional part and a two-dimensional part. The one-dimensional part is initialized with the initial probability distribution, and the two-dimensional part is a function representation of the feature vector pairs formed between the current point cloud and other point clouds. Therefore, when segmenting, not only the information of the current point cloud is considered, but also the relationship between the point clouds is considered, which is conducive to more complete segmentation of objects and backgrounds, and improves the effect of point cloud segmentation. When using the probability graph model to perform the inference calculation of the probability distribution of the point cloud category, due to the large calculation complexity of the original model, the present invention converts the original model into a simpler equivalent model based on the assumption that each random variable is independent of each other, which is conducive to reducing the amount of calculation, improving the calculation speed, and reducing the requirements for hardware. The method of the present invention reduces the search and fitting of feature points, is also conducive to the point cloud segmentation of complex special-shaped surface structures, and can adapt to the point cloud segmentation of textureless and low-texture objects, and can also adapt to changes in the external environment and illumination. Compared with the traditional model-based segmentation method, the method of the present invention does not require prior knowledge of the three-dimensional model of the object to be segmented, and has a wider application scenario.

[0153] Based on the above point cloud segmentation method, the present invention also provides a method for obtaining the position of an object, wherein the above point cloud segmentation method is used to improve the accuracy of object position determination. Figure 3 , a method for obtaining the position of an object in an embodiment includes steps 210 to 270, which are described in detail below.

[0154] Step 210: Obtain point cloud data of the object. The point cloud data of the object can be obtained by scanning the surface of one or more objects with a 3D scanner, or by obtaining a 2D image of the object surface with a 2D camera and then performing 3D reconstruction. The point cloud data represents the information of the points on the surface of the object. The point cloud data includes the 3D position information (x, y, z) of the point cloud, and may also include normal vectors, RGB color information, depth information, etc.

[0155] Step 220: Construct a feature vector of the point cloud using the point cloud data.

[0156] The feature vector can be constructed using one or more information in the point cloud data. In one embodiment, the feature vector can be constructed using the position information and normal vector of the point cloud to obtain a six-dimensional feature vector f i =(p i , n i ), where the subscript i represents the i-th point cloud, p i represents the location information of the i-th point cloud, n iRepresents the normal vector of the i-th point cloud. When the acquired point cloud sample data does not include the normal vector, normal vector estimation can also be performed, and the point cloud sample data can also be gridded.

[0157] Step 230: Initialize the probability that each point cloud belongs to each category, and obtain the initial probability distribution p of the point cloud category i =[p i1 , p i2 ,…,p ic ]. The category of a point cloud represents the object to which the point cloud belongs.

[0158] In one embodiment, a machine learning method can be used to classify point clouds to obtain the initial probability distribution of point cloud categories. First, the feature vector and category label of the point cloud sample are used to train the machine learning classification model. When performing point cloud segmentation, the feature vector of each point cloud is input into the trained machine learning classification model to obtain the probability that each point cloud belongs to each category, which is used as the initial probability distribution p of the point cloud category. i The machine learning classification model can be a neural network or SVM (Support Vector Machine), etc.

[0159] After obtaining the initial probability distribution p i After that, it can also be indexed and normalized, specifically: Get the initial probability distribution p i The maximum value in Calculate according to the following formula To replace the original probability value p ij :

[0160] Step 240: For each point cloud, the initial probability distribution p of the category of the point cloud i And the feature vector is input into the pre-trained probability graph model to obtain the probability distribution of the category of the point cloud. The probability graph model here can use the simpler new model described above.

[0161] Step 250: Take the point cloud category with the highest probability in the probability distribution as the category of the point cloud.

[0162] The above steps 210 to 250 may refer to steps 110 to 150 for details, which will not be described in detail here.

[0163] Step 260: According to the transformation matrix preset by the camera or scanner, the point cloud and its categories are mapped to an image, the image is subjected to connectivity analysis to obtain connected areas, and the connected areas are screened according to preset conditions, so that the area where each object is located is obtained. The screening can be performed according to the characteristics of the connected areas, and the screening characteristics can be area, length, width, roundness, etc. For example, the connected areas with an area less than a certain value or the connected areas with a roundness less than a certain value can be eliminated to reduce the interference of noise.

[0164] Step 270: Get the center of gravity of the area where each object is located, and map the area where the object is located and its center of gravity to the point cloud according to the transformation matrix. Then, the minimum enclosing rectangle and center of gravity of the object in space can be obtained. The length, width and height of the minimum enclosing rectangle are used as the size of the object, and the center of gravity is used to represent the position of the object.

[0165] The method for obtaining the position of an object according to the above-mentioned embodiment uses the point cloud segmentation method provided by the present invention, which can more completely segment the object and the background, improve the effect of point cloud segmentation, and at the same time reduce the amount of calculation and increase the calculation speed, so that the position of the object can be obtained more quickly and accurately.

[0166] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0167] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A method for point cloud segmentation, characterized in that: include: Acquire point cloud data of the object, wherein the point cloud data includes position information of the point cloud; constructing a feature vector of a point cloud using the point cloud data; Initialize the probability that each point cloud belongs to each category and obtain the initial probability distribution p of the point cloud category i =[p i1 ,p i2 ,…,p ic ], where i represents the i-th point cloud, p ij It represents the probability that the i-th point cloud belongs to the j-th category. The number of categories is c. The category of the point cloud represents the object to which the point cloud belongs. For each point cloud, the initial probability distribution p of the point cloud category is i The feature vector of the point cloud is input into a pre-trained probability graph model to obtain the probability distribution of the category of the point cloud, wherein the random variables in the probability graph model are the categories of the point cloud, and each random variable is independent of each other, and the energy function Q of the probability graph model is i (x i = l) includes a unary part and a binary part, the initial probability distribution pi is used to initialize the unary part, the binary part is a function representation of the feature vector pairs formed between the current point cloud and other point clouds, where l represents a certain category, and the energy function Q i (x i = l) represents the category x of the i-th point cloud i The probability of being l, the binary part is where l′ is a category other than l, μ(l,l′) is the label consistency term between category l′ and l, and w m is the weight, Q j (x j = l) represents the category x of the jth point cloud j The probability of being l, f i is the feature vector of the i-th point cloud, f j is the feature vector of the jth point cloud, h (m) (f i ,f j ) is the mth binary term, used to characterize the relationship between two point clouds; The point cloud segmentation is completed by taking the point with the highest probability in the probability distribution of each point cloud category as the category of this point cloud.

2. The method according to claim 1, characterized in that After completing the point cloud segmentation, it also includes: According to the preset transformation matrix, the point cloud and its categories are mapped to an image, the image is subjected to connectivity analysis to obtain connected regions, and the connected regions are screened according to preset conditions to obtain the region where each object is located; The center of gravity of the area where each object is located is obtained, and the area where the object is located and its center of gravity are mapped to the point cloud according to the transformation matrix to obtain the minimum circumscribed rectangle and center of gravity of the object in space.

3. The method according to claim 1 or 2, characterized in that The probabilistic graphical model is expressed as follows: in x i is a random variable representing the category of the i-th point cloud, Q i (x i ) is the probability distribution of the category of the i-th point cloud, l represents a certain category, l′ represents all other categories except l, μ(l,l′) is the label consistency term, w m is the weight, f i is the feature vector of the i-th point cloud, h (m) (f i ,f j ) is the mth binary term, used to characterize the relationship between two point clouds, K is the number of binary terms, is a unitary part and is the binary part, Z i is the normalization factor and 4. The method according to claim 3, characterized in that The probability graph model calculates the probability distribution of point cloud categories in the following way: First, Then iterate the following steps until Q i (x i )convergence: For all m, calculate calculate make Q i (x i ) are indexed and normalized.

5. The method according to claim 3, characterized in that The feature vector of the point cloud includes the position information and normal vector of the point cloud, K=2, and Among them, p i Represents the location information of the i-th point cloud, n i represents the normal vector of the i-th point cloud, σ1, σ2 and σ3 are preset parameters, and the eigenvector fi = (pi, ni) is a six-dimensional vector.

6. The method according to claim 4, characterized in that It also includes the initial probability distribution p i Perform indexation and normalization; Where pi or Q i (x i ) is indexed and normalized as follows: Get the maximum value in a probability distribution Calculate according to the following formula To replace the original probability value ζ ij : where ζ i =[ζ i1 ,ζ i2 ,…,ζ ic ] indicates p i or Q i (x i )’s probability distribution.

7. The method according to claim 3, characterized in that The probabilistic graphical model is trained in the following way: Get point cloud sample data and corresponding category labels; Construct the feature vector of the point cloud using the point cloud sample data; Initialize the probability that each point cloud belongs to each category and obtain the initial probability distribution p of the point cloud category i ′; The initial probability distribution p of the point cloud category i ′ and the feature vector of the point cloud are input into the probability graph model, and the weight w is updated using the gradient descent method according to the obtained results and category labels. m and the label consistency term μ(x i ,x j ), where the label consistency term μ(x i ,x j ) is specifically updated iteratively according to the following formula: in, The superscript n represents the nth iteration, λ is the preset learning rate, is the category label item of the i-th point cloud, when x i If the value is the same as the category label, it is 1, otherwise it is 0.

8. The method according to claim 7, characterized in that The probability of each point cloud belonging to each category is initialized to obtain the initial probability distribution p of the point cloud category i ',include: Set the initial probability of the same category as the class label to p e = -logP GT , the probability of other categories is set to p n = -log[(1-P GT ) / (c-1)], thereby obtaining the initial probability distribution p of the point cloud category i ′, where P GT is the preset confidence standard value.

9. The method according to claim 1 or 2, characterized in that: The probability of each point cloud belonging to each category is initialized to obtain the initial probability distribution p of the point cloud category i ,include: The feature vector of each point cloud is input into the trained machine learning classification model to obtain the probability that each point cloud belongs to each category, which is used as the initial probability distribution p of the point cloud category. i .

10. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the method according to any one of claims 1 to 9.

Images