A click-based interactive segmentation method for outdoor laser point clouds

Abstract

The present invention discloses an outdoor laser point cloud interactive segmentation method based on click, comprising the following steps: S1, representing the input 3D original point cloud as #imgabs0# where #imgabs1##imgabs2# are point p i 3D coordinates; S2, click the target of interest in the point cloud scene, divide the complete point cloud scene into small point cloud scenes containing the target of interest, and convert them into point cloud blocks containing the click position information; S3, generate the data required for training the network based on the computer simulation click method; S4, combine the segmentation loss function of the intersection ratio and the classification loss function of the cross entropy to form a hybrid loss function as the network loss function; S5, the convolution and pooling network CRSNet performs foreground and background segmentation on the point cloud blocks containing the click position information, and identifies the type of the target.

Description

Technical Field

[0001] The invention relates to the field of computer processing technology, and in particular to an outdoor laser point cloud interactive segmentation method based on clicking. Background Art

[0002] Interactive segmentation has become one of the hot research topics in recent years. It aims to obtain segmented objects of interest from interactive information such as strokes, bounding boxes, clicks, etc. These segmentation results selected by the user can be used for various downstream tasks, such as medical object detection, annotation of images and videos. Currently, interactive segmentation of point clouds is less explored. Nevertheless, it also plays an important role in some tasks, such as 3D building reconstruction, object localization and annotation. Therefore, new algorithms for interactive segmentation of point clouds are urgently needed.

[0003] Traditionally, interactive segmentation of point clouds is mainly achieved by using graph cut algorithms, including max-flow min-cut based algorithms, supervoxel and graph cut algorithms, interactive foreground extraction tools, and two-scale graph methods based on region-based graphs and pixel-level graphs. However, these graph cut-based methods can only capture low-level geometric features of point clouds and inevitably lose most of the high-level geometric information. In addition, the interactive behavior of strokes is usually inconvenient in complex scenes. Therefore, the performance of these existing traditional methods is far from satisfactory. In recent years, deep learning has made great progress in point cloud processing. Several effective deep learning-based methods have been proposed to capture point cloud deep features and achieve excellent performance. However, few deep learning-based interactive segmentation methods are used to process large amounts of mobile laser mapping point clouds.

[0004] In the past few decades, people have conducted a lot of research on interactive segmentation. Interactive segmentation methods for image regions can be roughly divided into two categories: methods based on handcrafted features and methods based on deep learning. In the early years, most methods were based on handcrafted feature methods such as the famous minimum cut maximum flow algorithm and GrabCut. Later, different techniques were proposed to improve the graph cut-based methods. However, these low-level feature-based methods cannot perfectly complete the object segmentation task in complex environments. Deep learning-based methods are different from traditional methods. Deep learning methods can extract high-level features from images and significantly improve the final segmentation performance. Existing point cloud methods are mainly based on graph cuts. Graph cut-based methods aim to calculate the minimum cut, which can divide the graph into two different parts. Converting point clouds into graphs is an important process in these algorithms.

[0005] Traditional clustering methods: Clustering methods are related to interactive segmentation because they can divide points into different groups, and with the help of interactive behavior, objects of interest can be easily classified into the same group. Traditional clustering methods include k-means, k-means++, seeded region growing, etc. They use low-level features instead of understanding the geometry of the target in space to make judgments, which causes the limitations of traditional methods. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide an outdoor laser point cloud interactive segmentation method based on click.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A click-based outdoor laser point cloud interactive segmentation method comprises the following steps:

[0009] S1. Represent the input 3D original point cloud as in It's point p i 3D coordinates of

[0010] S2, click on the target of interest in the point cloud scene, divide the complete point cloud scene into small point cloud scenes containing the target of interest, and convert them into point cloud blocks containing the click position information;

[0011] S3, generating data required for training the network based on a computer-simulated click method;

[0012] S4, combining the segmentation loss function of intersection-over-union ratio and the classification loss function of cross entropy to form a hybrid loss function as the loss function of the network;

[0013] S5. The convolution and pooling-based network CRSNet segments the foreground and background of the point cloud blocks containing the click location information and identifies the type of the target.

[0014] Preferably, the specific process of step S2 is as follows:

[0015] S21. The user clicks on the object to be segmented in the point cloud scene. The click is represented by 3D coordinates as c(x c ,y c ,z c ,);

[0016] S22, with the click as the center, a fixed scale σ as the range, and a top view perspective, the segmented point cloud blocks are segmented with P sIndicates that the value of σ is determined by the type of object to be segmented, cls indicates the type of object to be segmented, and the specific calculation formula for segmentation is expressed as:

[0017] P s ={p si (x i ,y i ,z i )||x i -x c |<σ,|y i -y c |<σ},

[0018]

[0019] S23, calculate point cloud block P s The distance d from all points to the click center c i , and transformed into a Gaussian vector g. The specific calculation formula is:

[0020] d i =||p si -c||2,

[0021]

[0022] S24, Gaussian vector g and point cloud block p s Stitched together, we get a point cloud block p containing the click location information s2 .

[0023] Preferably, the specific process of step S3 is as follows:

[0024] S31, input the 3D original point cloud data set containing object instance labels, the instance label information is represented as L = {l i ∈N|i=1,...,n}, where l i Indicates the instance category to which the i-th point belongs. The semantic label information is expressed as CLS = {cls i ∈{0,1}|i=1,...,n}, where cls i Indicates the type of object to which the i-th point belongs;

[0025] S32, using the instance label information contained in the data set, dividing the point cloud of the large scene into small point cloud scenes containing objects of interest;

[0026] S33, for each small point cloud scene P containing target information s , using the semantic label information CLS, obtain the target point cloud set

[0027] S34, for the target point cloud set P that needs to be segmentedo , the click center c is obtained using a computer simulation click method.

[0028] Preferably, the specific process of step S32 is as follows:

[0029] S321, inputting 3D original point cloud data including the point cloud of the target to be segmented;

[0030] S322, using instance label information to obtain the point set of the target to be segmented And project it onto the ground to get Calculate the center of the ground projection area The calculation formula is:

[0031]

[0032] S323, take the top view as the perspective, As the center, according to the scale σ of the target to be segmented, the small point cloud scene P containing the target of interest is obtained. s The specific calculation formula is as follows

[0033]

[0034]

[0035] Preferably, the click center c in step S34 can be implemented by using one of simulation method 1, simulation method 2, simulation method 3 or simulation method 4;

[0036] Simulation method 1: simulate clicking the location with the highest point density of the object, and use the mean-shift algorithm to find the point with the highest density as the click center c;

[0037] Simulation method 2: Simulate clicking the geometric center of the upper surface of the object. is the projection of the object point set on the ground, ε represents the noise, and the specific calculation formula is:

[0038]

[0039]

[0040] k = argmin(d i )

[0041] c=p k ;

[0042] Simulation method three: simulate clicking the geometric center of the object. The specific calculation formula is:

[0043] c={max(p i)+min(p i )} / 2+∈;

[0044] Simulation method 4: Click the center of mass of the object, and use the center of mass of the object as the click center. The specific calculation formula is:

[0045]

[0046]

[0047] k = argmin(d i )

[0048] c=p k .

[0049] Preferably, the specific process of step S4 is as follows:

[0050] S41. The true value label can be used with the segmentation label L s = {l s (i,j)|i=1,...,n;j=1,...,m} and a classification label L c = {l c (i,j)|i=1,...,n;j=1,...,cls}, where l s (i,j)∈{0,1}, 0 means that the jth point in the i-th point cloud scene belongs to the background, 1 means that it belongs to the object area that needs to be segmented, l c (i,j) indicates whether the object in the i-th point cloud scene belongs to the j-th category;

[0051] S42, segmentation loss function L based on intersection-over-union ratio IoU , the specific calculation formula is:

[0052]

[0053] Among them, p s (i,j) represents the probability that the jth point in the i-th point cloud scene is a point in the target area;

[0054] S43, classification loss function L based on cross entropy CE , the specific calculation formula is:

[0055]

[0056] Among them, q s (i,j) represents the probability that the jth point in the i-th point cloud scene is a background area point, p c (i,j) represents the probability that the object in the i-th point cloud scene belongs to the j-th class, B is the number of point cloud scenes processed by the network at one time, and N is the number of points contained in each point cloud scene;

[0057] S44, mixed loss function L mix , the specific calculation formula is:

[0058] L mix =αL IoU +βL CE

[0059] Among them, α and β are constant parameters.

[0060] Preferably, the specific process of step S5 is as follows:

[0061] S51, establishing a CRSNet network structure including a feature extraction module, a classification module and a segmentation module;

[0062] S52, feature extraction module, used to extract point cloud features, the input data passes through multiple convolution perception layers and maximum pooling layers in sequence to obtain a 1x1024 vector;

[0063] S53, a classification module, is used to obtain a 1xk vector from the 1x1024 vector obtained from the feature extraction module through multiple layers of fully connected layers;

[0064] S54, segmentation module, is used to obtain the 1x1024 vector from the feature extraction module, pass through multiple layers of fully connected layers, obtain a 1x128 vector, copy the nx128 matrix, and concatenate it with the nx1024 matrix in the feature extraction module, and then pass through multiple layers of convolutional perception layers to obtain the output nx2.

[0065] After adopting the above technical solution, the present invention has the following advantages compared with the background technology:

[0066] 1. The present invention provides a click-based interactive segmentation method for outdoor laser point clouds, which uses a newly designed interactive segmentation network to process large-scale outdoor point clouds and can provide reliable instance segmentation results in real time.

[0067] 2. The present invention discloses a click-based outdoor laser point cloud interactive segmentation method, which adopts an effective interactive mode and interactive information fusion method. Only one click is required to quickly and accurately obtain the object to be segmented.

[0068] 3. The present invention provides an outdoor laser point cloud interactive segmentation method based on clicking, and proposes four algorithms for computer simulation of human clicking on objects, which can quickly convert a data set containing only instance information into a data set containing interactive information.

[0069] 4. The present invention provides a click-based outdoor laser point cloud interactive segmentation method, which adopts ChebyNet as the basic network and proposes a graph convolutional network based on skip connections, thereby effectively improving the efficiency of target classification based on three-dimensional point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the process of the present invention;

[0071] Figure 2 It is a framework diagram of the CRSNet network of the present invention;

[0072] Figure 3 A schematic diagram of the present invention using clicks for rough segmentation;

[0073] Figure 4 Schematic diagram of the conversion of click position and Gaussian vector used in the present invention; (a) calculating the distance from the midpoint of the point cloud block to the click center; (b) converting the distance into a Gaussian vector; (c) schematic diagram of a sample after rough segmentation. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0075] A click-based outdoor laser point cloud interactive segmentation method comprises the following steps:

[0076] S1. Represent the input 3D original point cloud as in It's point p i 3D coordinates of

[0077] S2, click on the target of interest in the point cloud scene, divide the complete point cloud scene into small point cloud scenes containing the target of interest, and convert them into point cloud blocks containing the click position information;

[0078] The specific process of step S2 is as follows:

[0079] S21. The user clicks on the object to be segmented in the point cloud scene. The click is represented by 3D coordinates as c(x c ,y c ,z c ,);

[0080] S22, with the click as the center, a fixed scale σ as the range, and a top view perspective, the segmented point cloud blocks are segmented with P sIndicates that the value of σ is determined by the type of object to be segmented, cls indicates the type of object to be segmented, and the specific calculation formula for segmentation is expressed as:

[0081] P s ={p si (x i ,y i ,z i )||x i -x c |<σ,|y i -y c |<σ},

[0082]

[0083] S23, calculate point cloud block P s The distance d from all points to the click center c i , and transformed into a Gaussian vector g. The specific calculation formula is:

[0084] d i =||p si -c||2,

[0085]

[0086] S24, Gaussian vector g and point cloud block p s Stitched together, we get a point cloud block p containing the click location information s2 ;

[0087] S3, generating data required for training the network based on a computer-simulated click method;

[0088] The specific process of step S3 is as follows:

[0089] S31, input the 3D original point cloud data set containing object instance labels, the instance label information is represented as L = {l i ∈N|i=1,...,n}, where l i Indicates the instance category to which the i-th point belongs. The semantic label information is expressed as CLS = {cls i ∈{0,1}|i=1,...,n}, where cls i Indicates the type of object to which the i-th point belongs;

[0090] S32, using the instance label information contained in the data set, dividing the point cloud of the large scene into small point cloud scenes containing objects of interest;

[0091] The specific process of step S32 is as follows:

[0092] S321, inputting 3D original point cloud data including the point cloud of the target to be segmented;

[0093] S322, using instance label information to obtain the point set of the target to be segmented And project it onto the ground to get Calculate the center of the ground projection area The calculation formula is:

[0094]

[0095] S323, take the top view as the perspective, As the center, according to the scale σ of the target to be segmented, the small point cloud scene P containing the target of interest is obtained. s The specific calculation formula is as follows

[0096]

[0097]

[0098] S33, for each small point cloud scene P containing target information s , using the semantic label information CLS, obtain the target point cloud set

[0099] S34, for the target point cloud set P that needs to be segmented o , using computer simulation click method to obtain click center c;

[0100] The click center c in step S34 may be implemented by using one of simulation method 1, simulation method 2, simulation method 3 or simulation method 4;

[0101] Simulation method 1: simulate clicking the location with the highest point density of the object, and use the mean-shift algorithm to find the point with the highest density as the click center c;

[0102] Simulation method 2: Simulate clicking the geometric center of the upper surface of the object. is the projection of the object point set on the ground, ε represents the noise, and the specific calculation formula is:

[0103]

[0104]

[0105] k = argmin(d i )

[0106] c=p k ;

[0107] Simulation method three: simulate clicking the geometric center of the object. The specific calculation formula is:

[0108] c={max(p i )+min(p i )} / 2+∈;

[0109] Simulation method 4: Click the center of mass of the object, and use the center of mass of the object as the click center. The specific calculation formula is:

[0110]

[0111]

[0112] k = argmin(d i )

[0113] c=p k ;

[0114] S4, combining the segmentation loss function of intersection-over-union ratio and the classification loss function of cross entropy to form a hybrid loss function as the loss function of the network;

[0115] The specific process of step S4 is as follows:

[0116] S41. The true value label can be used with the segmentation label L s = {l s (i,j)|i=1,...,n;j=1,...,m} and a classification label L c = {l c (i,j)|i=1,...,n;j=1,...,cls}, where l s (i,j)∈{0,1}, 0 means that the jth point in the i-th point cloud scene belongs to the background, 1 means that it belongs to the object area that needs to be segmented, l c (i,j) indicates whether the object in the i-th point cloud scene belongs to the j-th category;

[0117] S42, segmentation loss function L based on intersection-over-union ratio IoU , the specific calculation formula is:

[0118]

[0119] Among them, p s (i,j) represents the probability that the jth point in the i-th point cloud scene is a point in the target area;

[0120] S43, classification loss function L based on cross entropy CE , the specific calculation formula is:

[0121]

[0122] Among them, q s (i,j) represents the probability that the jth point in the i-th point cloud scene is a background area point, p c (i,j) represents the probability that the object in the i-th point cloud scene belongs to the j-th class, B is the number of point cloud scenes processed by the network at one time, and N is the number of points contained in each point cloud scene;

[0123] S44, mixed loss function L mix , the specific calculation formula is:

[0124] L mix =αL IoU +βL CE

[0125] Among them, α and β are constant parameters;

[0126] S5, based on the convolution and pooling network CRSNet, the point cloud block containing the click location information is segmented into foreground and background, and the type of the target is identified;

[0127] The specific process of step S5 is as follows:

[0128] S51, establishing a CRSNet network structure including a feature extraction module, a classification module and a segmentation module;

[0129] S52, feature extraction module, used to extract point cloud features, the input data passes through multiple convolution perception layers and maximum pooling layers in sequence to obtain a 1x1024 vector;

[0130] S53, a classification module, is used to obtain a 1xk vector from the 1x1024 vector obtained from the feature extraction module through multiple layers of fully connected layers;

[0131] S54, segmentation module, is used to obtain the 1x1024 vector from the feature extraction module, pass through multiple layers of fully connected layers, obtain a 1x128 vector, copy the nx128 matrix, and concatenate it with the nx1024 matrix in the feature extraction module, and then pass through multiple layers of convolutional perception layers to obtain the output nx2.

[0132] Example

[0133] The flow chart of this embodiment can be seen Figure 1 The specific operation process is as follows:

[0134] 1. Collect data and label a small amount of it as training samples

[0135] Use a mobile laser scanning system to obtain a road object dataset in an urban road environment. This dataset contains various road objects such as vehicles, pedestrians, and bicycles. Manually annotate a small part of the data to obtain instance labels, semantic labels, and interactive information of user clicks. Alternatively, collect public point cloud road datasets such as KITT I on the Internet as training samples.

[0136] 2. Add interaction information to the labeled data set as training data samples

[0137] For the data set that already contains interactive information, it is used as a sample data set for network training after the interactive information processing step. Figure 3 It shows how a large scene can be segmented into small blocks of point clouds and Gaussian vectors by interactive clicking and combined into new point cloud blocks. Figure 4 The specific calculation method of Gaussian vector is shown. For a data set without interactive information, the method of using a computer to simulate user operations proposed by the present invention is used to obtain simulated interactive clicks, and then Figure 3 , Figure 4 way to obtain point cloud blocks containing click location information that can be processed by the network.

[0138] 3. Adjust the parameters of the network

[0139] Use the scale information, density information, scene size, etc. of the labeled objects to set the parameters required by the network. Specifically, set the parameters of the loss function, the batch size of the network, the number of point cloud points, and finally use the prepared training data set to train the network model.

[0140] 4. Use the trained network to label the remaining unlabeled data

[0141] By using the trained network model, using a mouse or other tool to click on the objects to be labeled, and then sending them to the network, the required segmentation results can be obtained. In order to quickly verify the performance of the network, the computer simulation human clicking algorithm proposed in the present invention can also be used to replace real people's object clicking.

[0142] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A click-based interactive segmentation method for outdoor laser point clouds, characterized by: The following steps are involved: S1. Represent the input 3D original point cloud as in It's point p i 3D coordinates of S2, click on the target of interest in the point cloud scene, divide the complete point cloud scene into small point cloud scenes containing the target of interest, and convert them into point cloud blocks containing the click position information; The specific process of step S2 is as follows: S21. The user clicks on the object to be segmented in the point cloud scene. The click is represented by 3D coordinates as c(x c ,y c ,z c ); S22, with the click as the center, a fixed scale σ as the range, and a top view perspective, the segmented point cloud blocks are segmented with P s Indicates that the value of σ is determined by the type of object to be segmented, cls indicates the type of object to be segmented, and the specific calculation formula for segmentation is expressed as: P s ={p si (x i ,y i ,z i )|||x i -x c |<σ,|y i -y c |<σ}, S23, calculate point cloud block P s The distance d from all points to the click center c i , and transformed into a Gaussian vector g. The specific calculation formula is: d i =||p si -c||2, S24, Gaussian vector g and point cloud block p s Stitched together, we get a point cloud block p containing the click location information s2 ; S3, generating data required for training the network based on a computer-simulated click method; The specific process of step S3 is as follows: S31, input the 3D original point cloud data set containing object instance labels, the instance label information is represented as L = {l i ∈N|i=1,...,n}, where l i Indicates the instance category to which the i-th point belongs. The semantic label information is expressed as CLS = {cls i ∈{0,1}|i=1,...,n}, where cls i Indicates the type of object to which the i-th point belongs; S32, using the instance label information contained in the data set, dividing the point cloud of the large scene into small point cloud scenes containing objects of interest; The specific process of step S32 is as follows: S321, inputting 3D original point cloud data including the point cloud of the target to be segmented; S322, using instance label information to obtain the point set of the target to be segmented And project it onto the ground to get Calculate the center of the ground projection area The calculation formula is: S323, take the top view as the perspective, As the center, according to the scale σ of the target to be segmented, the small point cloud scene P containing the target of interest is obtained. s The specific calculation formula is as follows S33, for each small point cloud scene P containing target information s , using the semantic label information CLS, obtain the target point cloud set S34, for the target point cloud set P that needs to be segmented o , using computer simulation click method to obtain click center c; The click center c in step S34 may be implemented by using one of simulation method 1, simulation method 2, simulation method 3 or simulation method 4; Simulation method 1: simulate clicking the location with the highest point density of the object, and use the mean-shift algorithm to find the point with the highest density as the click center c; Simulation method 2: Simulate clicking the geometric center of the upper surface of the object. is the projection of the object point set on the ground, ε represents the noise, and the specific calculation formula is: <h2 style=";text-align:left;direction:ltr">k = argmin(d)<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ) c=p k ; Simulation method three: simulate clicking the geometric center of the object. The specific calculation formula is: c={max(p i )+min(p i )} / 2+∈; Simulation method 4: Click the center of mass of the object, and use the center of mass of the object as the click center. The specific calculation formula is: <h2 style=";text-align:left;direction:ltr">k = argmin(d)<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ) c=p k ; S4, combining the segmentation loss function of intersection-over-union ratio and the classification loss function of cross entropy to form a hybrid loss function as the loss function of the network; The specific process of step S4 is as follows: S41. The true value label can be used with the segmentation label L s = {l s (i,j)|i=1,…,n;j=1,…,m} and a classification label L c = {l c (i,j)|i=1,…,n;j=1,…,cls}, where l s (i,j)∈{0,1}, 0 means that the jth point in the i-th point cloud scene belongs to the background, 1 means that it belongs to the object area that needs to be segmented, l c (i,j) indicates whether the object in the i-th point cloud scene belongs to the j-th category; S42, segmentation loss function L based on intersection-over-union ratio IoU , the specific calculation formula is: Among them, p s (i,j) represents the probability that the jth point in the i-th point cloud scene is a point in the target area; S43, classification loss function L based on cross entropy CE , the specific calculation formula is: Among them, q s (i,j) represents the probability that the jth point in the i-th point cloud scene is a background area point, p c (i,j) represents the probability that the object in the i-th point cloud scene belongs to the j-th class, B is the number of point cloud scenes processed by the network at one time, and N is the number of points contained in each point cloud scene; S44, mixed loss function L mix , the specific calculation formula is: L mix =αL IoU +βL CE Among them, α and β are constant parameters; S5, based on the convolution and pooling network CRSNet, the point cloud block containing the click location information is segmented into foreground and background, and the type of the target is identified; The specific process of step S5 is as follows: S51, establishing a CRSNet network structure including a feature extraction module, a classification module and a segmentation module; S52, feature extraction module, used to extract point cloud features, the input data passes through multiple convolution perception layers and maximum pooling layers in sequence to obtain a 1x1024 vector; S53, a classification module, is used to obtain a 1xk vector from the 1x1024 vector obtained from the feature extraction module through multiple layers of fully connected layers; S54, segmentation module, is used to obtain the 1x1024 vector from the feature extraction module, pass through multiple layers of fully connected layers, obtain a 1x128 vector, copy the nx128 matrix, and concatenate it with the nx1024 matrix in the feature extraction module, and then pass through multiple layers of convolutional perception layers to obtain the output nx2.

Images