Multi-feature fusion and graph optimization unsupervised point cloud segmentation method and system

Abstract

The invention discloses an unsupervised point cloud segmentation method and system based on multi-feature fusion and graph optimization. The method comprises the following steps: training a point cloud instance segmentation model by using a 3D indoor scene data set; firstly, scene point cloud in a data set is extracted to segment a foreground and a background, downsampling is carried out on the foreground, and features are extracted; then constructing an undirected graph segmentation point cloud distribution pseudo tag, and performing up-sampling to a complete point cloud; inputting a Mask3D model, and training a segmentation model in combination with a weak supervision loss function; and finally, deploying a depth camera to capture a target scene point cloud, and inputting the trained model to output an instance segmentation result. According to the method, the traditional point features and the pre-training features are considered, the performance dependence on the pre-training model is reduced, and the reliability of the method is improved. Meanwhile, aiming at the characteristics that foreground objects in an indoor scene are diverse in distribution and complex in structure, a foreground separation method is innovatively adopted, so that the model can be adjusted and optimized aiming at a foreground effect, and a better segmentation effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer graphics, three-dimensional point cloud, computer vision, and in particular to a multi-feature fusion and graph optimization unsupervised point cloud segmentation method and system. BACKGROUND

[0002] In the field of computer vision, three-dimensional vision is an important branch. Three-dimensional vision refers to the ability to perceive, understand and analyze objects and scenes in three-dimensional space through computer technology. Point cloud is a more mainstream data type. Point cloud has the characteristics of disorder, sparsity and non-structured, and its data is huge, and the annotation work is tedious, cumbersome and expensive, such as the commonly used dataset Scannet, which takes about 22.3 minutes to annotate each scene. Therefore, high-quality point cloud datasets are very scarce.

[0003] In order to solve this problem, various point cloud unsupervised segmentation works have been proposed, such as the early SuperVoxel method based on geometric features. The recently proposed unsupervised learning method includes the PointContrast method, but this work still relies on a considerable number of frame, point annotations or a certain proportion of mask annotations to achieve good results. The PointDC model needs to rely on projection and 2D pre-training model to preliminarily obtain point features, and the Unscene3D method uses 2D and 3D models to process color and geometric features respectively. These methods are very dependent on the performance of the pre-training model in terms of accuracy, and there is still a lot of room for improvement.

[0004] One of the current prior arts is the PointDC model proposed in the paper "Pointdc: Unsupervised semantic segmentation of 3d point clouds via cross-modal distillation and super-voxel clustering". The model is an unsupervised point cloud segmentation model. It includes two stages: cross-modal distillation (CMD) and super-voxel clustering (SVC). In the first stage of CMD, multi-view visual features are back-projected to 3D space and aggregated into unified point features to distill point representation training. In the second stage of SVC, point features are aggregated into super-voxels and then input into an iterative clustering process to mine semantic classes. The disadvantage of this technology is that it needs to perform multiple view projection and back-projection operations, and relies on the performance of the 2D image segmentation model. At the same time, the model needs a large number of iterations and needs a relatively long training time.

[0005] The second prior art is the UnScene3D method proposed in the paper "Unscene3d: Unsupervised 3d instance segmentation for indoor scenes". This method is a completely unsupervised 3D instance segmentation method designed for complex indoor scenes. Its core process is divided into two steps: (1) initial pseudo mask generation, first decompose the common into geometric primitives, and then use 2D and 3D models to extract 2D color features and 3D geometric features, and use normalized graph cut to generate the initial pseudo mask. (2) Self-training optimization: train the 3D model based on the pseudo mask, and combine DiceLoss+BCEloss to train the model iteratively to improve the model accuracy. The disadvantage of this technology is the lack of ability to divide the scene objects (foreground) from the background, and the initial pseudo mask does not consider the coordinates and color information of the points itself, and its quality depends on the performance of the self-supervised model itself. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the existing method, and a multi-feature fusion and graph optimization unsupervised point cloud segmentation method and system are proposed. The main problems solved by the present application are: first, the coordinates and color features of the points themselves are introduced to reduce the dependence on the performance of the pre-training model and reduce the impact of poor pre-training performance; second, the problem of foreground and background mixing is solved.

[0007] In order to solve the above problems, the present application proposes a multi-feature fusion and graph optimization unsupervised point cloud segmentation method, which comprises:

[0008] Using a 3D indoor scene dataset as training data, for any data item in the training data, extracting the scene point cloud V full containing coordinate information and color information;

[0009] Segmenting the scene point cloud V full into foreground point cloud V fg and background point cloud V bg , then performing farthest point sampling on the foreground point cloud V fg to obtain a down-sampled point cloud V sample , and then inputting the down-sampled point cloud V sample into a pre-trained model φ to extract deep learning features F, and the coordinate information of V sample is denoted as P, and the color information is denoted as C;

[0010] Using the down-sampled point cloud V sample to construct an undirected graph G={V,E,A} containing points V, edges E, and edge affinity A, and then using a fast multi-cut algorithm to solve the minimum segmentation problem to form V samplesegmentation result, the segmentation result contains a plurality of point sets, and then an initial pseudo label is assigned to each point set, and a label set is denoted as L sampled ;

[0011] The L sampled is up-sampled to the foreground point cloud V fg to obtain a foreground label, and the background point cloud V bg is assigned a background label; the foreground label and the background label together form a complete pseudo label L of the scene point cloud V full ; a model used for generating the complete pseudo label is denoted as a point cloud segmentation model;

[0012] The scene point cloud V full , the complete pseudo label L, is input into the Mask3D segmentation model, and a weakly supervised loss function composed of a coordinate center loss, a bounding box loss, a cross-entropy loss and a Dice loss is constructed, the point cloud segmentation model is trained through two steps of over-segmentation training and under-segmentation optimization, and a trained point cloud instance segmentation model is formed;

[0013] For a target scene to be segmented, a user uses a depth camera deployed at different positions in the target scene to take pictures, obtains a depth image including color information and depth information, then calculates point cloud coordinates using the depth information, calculates point cloud colors using the color information, and obtains a scene point cloud V target containing coordinate information and color information, and then inputs the trained point cloud instance segmentation model to obtain an instance segmentation result of the point cloud.

[0014] Preferably, the 3D indoor scene dataset includes ScanNet and S3DIS.

[0015] Preferably, the scene point cloud V full is segmented into a foreground point cloud V fg and a background point cloud V bg , then the foreground point cloud V fg is subjected to farthest point sampling to obtain a down-sampled point cloud V sample , and then the down-sampled point cloud V sample is input into a pre-trained model φ to extract a deep learning feature F, specifically as follows:

[0016] First, the scene point cloud V full is subjected to background separation:

[0017] First, the extreme values in each direction are used to determine a bounding box, and the surface normal vector of the bounding box is used as the normal vector of a candidate plane; the candidate plane is represented as where i is the subscript of the surface, and q is a column vector [x, y, z] T , n iis the normal vector of each surface, d i is the intercept;

[0018] For each candidate plane, calculate the intersection over union (IOU) of its projection with each bounding box surface,

[0019]

[0020] where Area represents the projected area, H i is the set of points in the plane, is the orthogonal projection function, B i is the bounding box surface; when the intersection over union is greater than a pre-set threshold, it is removed as background (such as walls, floors) in the scene, and the foreground point cloud is retained; at this point, the scene point cloud V full is divided into foreground point cloud V fg and background point cloud V bg .

[0021] Secondly, the farthest point sampling is performed on the foreground point cloud V fg :

[0022] An initial point S1 is randomly selected, and the remaining points are represented as R1 = V fg \S1; at the k-1 step of the iteration process, S k-1 represents the current sampled point set, and the newly selected sampling point S k-1 ′ is determined by finding the point p in the unsampled point set R k-1 that maximizes the distance function D(p, S k-1 ), i.e., Distance function D(p, S k-1 ) = min ||p xyz ,s xyz ||2, where p xyz and s xyz represent the three-dimensional coordinates of the unsampled points and the sampled points, respectively; then update the set S k = S k-1 ′∪S k-1 , R k = R k-1 \S k and enter the next iteration step;

[0023] After the iteration is completed, the set S k is taken as the result of the downsampling of the foreground point cloud V fg , denoted as V sample .

[0024] Thirdly, the deep learning features are extracted:

[0025] The V sampleInput a pre-trained model φ (PointTransfromer model is adopted specifically), to obtain a deep feature F = φ (V sample ).

[0026] Preferably, the down-sampled point cloud V sample is used to construct an undirected graph G = {V, E, A} containing points V, edges E and edge affinity A, and then a fast multi-cut algorithm is used to solve the minimum segmentation problem to form a segmentation result of V sample , which contains multiple point sets, and then an initial pseudo label is assigned to each point set, and the label set is denoted as L sampled , specifically:

[0027] Firstly, a point cloud graph is constructed:

[0028] When the down-sampled point cloud V sample is used to construct an undirected graph G = {V, E, A}, a k-nearest neighbor method is used to find the neighboring points of each point, and edges E are constructed with these neighboring points.

[0029] The edge affinity A is composed of the difference values of the depth feature, coordinates and color: for the nodes with subscripts i and j in the graph, the cosine similarity is used to calculate the depth feature affinity A (i,j)emb = Cos (F i , F j ), where F is the depth feature of V sample ; the L2 distance is used to calculate the coordinate affinity A (i,j)pos = -||P i , P j || and the color affinity A (i,j)color = -||C i , C j ||, where P and C are the coordinate information and color information of V sample ; the final edge affinity A ij = α1A (i,j)emb + α2A (i,j)pos + α3A (i,j)color , where α1, α2 and α3 are hyperparameters.

[0030] Secondly, the down-sampled point cloud V sample is segmented and labeled:

[0031] The fast multi-cut algorithm RAMA is used to form a minimum segmentation graph by solving the following optimization problem:

[0032]

[0033] where x ij ∈ {0, 1} represents whether to cut the edge (i, j), representing node v i to node v j a set of all paths, (u, v) represents path Path i→j two endpoints of a segment of edges in Path (i,j)∈E A ij x ij , the constraint condition is that if two nodes v i , v j belong to the same subgraph after partition, then all edges on the path of the two nodes are not cut x uv = 0, that is, to ensure the connectivity of the subgraph; after partitioning into multiple point sets, an initial pseudo label is assigned to each point set, and the label set is denoted as L sampled .

[0034] Preferably, the L sampled is up-sampled to the foreground point cloud V fg to obtain a foreground label, and the background point cloud V bg is assigned a background label; the foreground label and the background label together form a complete pseudo label L of the scene point cloud V full , specifically:

[0035] The L sampled is restored to the size of the V fg by a nearest neighbor search: first, the k-nearest neighbor method is used to find the labels of the k-nearest points of each point in the V fg in the V sample , and then a majority vote is performed on the k points, and the formula is:

[0036]

[0037] wherein LU represents a set of candidate labels, l represents a label l selected from the label set LU, N k represents a set of subscripts 1 to k, and I represents 1 if the bracket is true, and 0 otherwise; represents a predicted label, l i represents the labels of the k neighboring points; after calculation, a predicted label of each foreground point set is obtained, and the set of the predicted labels is a foreground label; then, the points in the V bg are taken as background, and a background label is assigned by using a majority vote method; the foreground label and the background label together form a complete pseudo label L of the scene point cloud V full .

[0038] Preferably, the weakly supervised loss function composed of a coordinate center loss, a bounding box loss, a cross-entropy loss and a Dice loss is constructed, and the weakly supervised loss function is specifically:

[0039] For the coordinate center, the training target is to minimize the Euclidean distance between the predicted coordinate center and the real coordinate center, and the loss value is represented as:

[0040] L mean = Euclidean (avg (m), avg (m * ))

[0041] where m is the real point set, m * is the predicted point set, and avg is the average value;

[0042] For the bounding box, the loss of the bounding box is calculated by the Euclidean distance between the maximum and minimum values of the two point sets, and the loss value is represented as:

[0043] L box = sum (Euclidean (max (m), max (m * )), Euclidean (min (m), min (m * ))

[0044] In addition, there are cross-entropy loss L BCE and Dice loss L dice , represented as:

[0045]

[0046] where lt i is the real label, is the predicted label, and N is the number of points of the scene point cloud V full ;

[0047] The final loss is:

[0048] L = λ1L dice + λ2L BCE + λ3L mean + λ4L box

[0049] where λ1, λ2, λ3, λ4 are weight parameters.

[0050] Preferably, for the target scene to be segmented, the user uses a depth camera deployed at different positions in the target scene to take pictures, obtaining a depth image including color information and depth information, then calculates the point cloud coordinates using the depth information, calculates the point cloud color using the color information, and obtains a scene point cloud V target containing coordinate information and color information, then inputs the trained point cloud instance segmentation model to obtain the instance segmentation result of the point cloud, specifically:

[0051] Prepare three depth cameras in three directions, and deploy them to different positions of the scene; after the deployment of the devices is completed, record the coordinates [x0, y0, z0] of each depth camera; use the deployed depth camera to take pictures, and obtain a plurality of depth images I depth The depth image contains RGB-D information, and the RGB-D information refers to that one pixel in the picture has RGB color information and D depth information; Gaussian filtering is applied to each depth image to remove noise;

[0052] Then, the position of the point cloud is calculated through the depth information and the picture position, and the mapping relationship between the pixel and the corresponding point cloud is saved; first, the relative coordinates x c ,y c ,z c of the camera are calculated

[0053]

[0054] Wherein, u, v represent any coordinate points in the image, u0, v0 represent the center of the image, d is the depth information, f is the focal length of the camera, x0, y0, z0 are the coordinates of the camera, dx represents the physical size of the pixel in the x-axis direction, and dy represents the physical size of the pixel in the y-axis direction;

[0055] Then, the unified world coordinates [x w ,y w ,z w ] are calculated according to the relative coordinates T =R·[x c ,y c ,z c ] T +[x0, y0, z0], wherein R is a rotation matrix;

[0056] For the color information of the point cloud, the average value of each depth picture is calculated , wherein n is the total number of depth pictures, and Ci is the RGB value of the i-th picture;

[0057] Finally, the scene point cloud V target of the target scene to be segmented including coordinate information and color information is obtained target , and the V target is input into the trained point cloud instance segmentation model to obtain the instance segmentation result of the point cloud.

[0058] Correspondingly, the present application also provides an unsupervised point cloud segmentation system based on multi-feature fusion and graph optimization, comprising:

[0059] A point cloud information extraction unit is configured to use a 3D indoor scene dataset as training data, and for any data item in the training data, extract a scene point cloud V full; the scene point cloud V full is segmented into a foreground point cloud V fg and a background point cloud V bg , then the foreground point cloud V fg is subjected to farthest point sampling to obtain a down-sampled point cloud V sample , and then the down-sampled point cloud V sample is input into a pre-trained model φ to extract a deep learning feature F, V sample , wherein the coordinate information of V sample is recorded as P, and the color information is recorded as C;

[0060] A sampling label generation unit is configured to construct an undirected graph G = {V, E, A} containing points V, edges E and edge affinity A by using the down-sampled point cloud V sample , and then solve a minimum segmentation problem by cooperating with a fast multi-cut algorithm to form a segmentation result of V sample , wherein the segmentation result contains multiple point sets, and then an initial pseudo label is assigned to each point set, and a label set is recorded as L sampled ;

[0061] A complete label generation unit is configured to up-sample the L sampled to the foreground point cloud V fg to obtain a foreground label, and assign a background label to the background point cloud V bg ; the foreground label and the background label jointly form a complete pseudo label L of the scene point cloud V full ; a model used for generating the complete pseudo label is recorded as a point cloud segmentation model;

[0062] A model training unit is configured to input the scene point cloud V full and the complete pseudo label L, use a Mask3D segmentation model, and construct a weakly supervised loss function composed of a coordinate center loss, a bounding box loss, a cross-entropy loss and a Dice loss, train the point cloud segmentation model through two steps of over-segmentation training and under-segmentation optimization, and form a trained point cloud instance segmentation model;

[0063] A deployment application unit is configured to, for a target scene to be segmented, use a depth camera deployed at different positions in the target scene to take pictures to obtain a depth image containing color information and depth information, then calculate point cloud coordinates by using the depth information, calculate point cloud colors by using the color information, and obtain a scene point cloud V target containing coordinate information and color information, and then input the trained point cloud instance segmentation model to obtain an instance segmentation result of the point cloud.

[0064] The present application has the following beneficial effects:

[0065] This invention employs a novel graph construction and allocation method for unsupervised learning of point clouds, addressing the issues of scarce and difficult-to-label point cloud datasets. Compared to other unsupervised learning methods, this invention combines traditional point features with pre-trained features, reducing reliance on the performance of pre-trained models and improving the reliability of the method. Furthermore, considering the diverse distribution and complex structure of foreground objects in indoor scenes, this invention innovatively adopts a foreground separation method, allowing the model to be optimized for foreground effects and achieve better segmentation results. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the overall process of an unsupervised point cloud segmentation method based on multi-feature fusion and graph optimization according to an embodiment of the present invention.

[0067] Figure 2 This is a structural diagram of an unsupervised point cloud segmentation system for multi-feature fusion and graph optimization according to an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Figure 1 This is a flowchart illustrating the overall process of an unsupervised point cloud segmentation method based on multi-feature fusion and graph optimization according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0070] S1, using a 3D indoor scene dataset as training data, for any data item in the training data, extract the scene point cloud V containing coordinate and color information. full ;

[0071] S2, the scene point cloud V full Divided into foreground viewpoints and cloud V fg and background point cloud V bg Then, for the previous scenic spot, Cloud V fg Perform farthest point sampling to obtain downsampled point cloud V sample Then downsample the point cloud V sample Deep learning features F and V are extracted from the pre-trained model φ. sample The coordinate information is denoted as P, and the color information is denoted as C;

[0072] S3, using the downsampled point cloud V sampleConstruct an undirected graph G = {V, E, A} containing points V, edges E, and edge affinity A, and then solve the minimum cut problem with a fast multi-cut algorithm to form V sample , a segmentation result containing multiple point sets, and then assign an initial pseudo-label to each point set, and the label set is denoted as L sampled .

[0073] S4, upsample the L sampled to the foreground point cloud V fg to obtain foreground labels, and assign background labels to the background point cloud V bg ; the foreground labels and the background labels together form the complete pseudo-label L of the scene point cloud V full ; the model used to generate the complete pseudo-label is denoted as a point cloud segmentation model.

[0074] S5, input the scene point cloud V full , the complete pseudo-label L, use the Mask3D segmentation model, and construct a weakly supervised loss function composed of coordinate center loss, bounding box loss, cross-entropy loss, and Dice loss, train the point cloud segmentation model through two steps of over-segmentation training and under-segmentation optimization, and form a trained point cloud instance segmentation model.

[0075] S6, for a target scene to be segmented, a user uses a depth camera deployed at different positions in the target scene to take pictures, obtains a depth image including color information and depth information, then calculates point cloud coordinates using the depth information and point cloud colors using the color information, obtains a scene point cloud V target containing coordinate information and color information, and then inputs the trained point cloud instance segmentation model to obtain an instance segmentation result of the point cloud.

[0076] Step S1 is as follows:

[0077] The source of the 3D indoor scene data includes ScanNet and S3DIS datasets. For any data item in the dataset, extract the scene point cloud V full containing coordinate information and color information.

[0078] Step S2 is as follows:

[0079] S2-1, perform background separation on the scene point cloud V full : For indoor point cloud datasets, the background includes the floor, walls, and ceiling, which are usually large and flat surfaces, and thus can be easily removed.

[0080] Based on the Manhattan world assumption, the world is considered as a structure composed of axes intersecting at right angles, which is also true for indoor scenes. Therefore, bounding boxes are first determined by the extreme values of each direction, and the surface normal vectors of the bounding boxes are taken as the normal vectors of the candidate planes; the candidate plane is represented as where i is the subscript of the surface, q is the column vector [x, y, z] T , n i is the normal vector of each surface, and d i is the intercept;

[0081] For each candidate plane, the intersection-over-union (IOU) of its projection with each bounding box surface is calculated,

[0082]

[0083] where Area represents the area of the projection region, H i is the point set in the plane, is the orthogonal projection function, and B i is the bounding box surface; when the intersection-over-union is greater than a pre-set threshold, it is removed as the background (such as walls and floors) in the scene, and the foreground point cloud is retained; at this point, the scene point cloud V full is divided into foreground point cloud V fg and background point cloud V bg .

[0084] S2-2, in order to reduce the computational complexity and enhance the sparsity of the feature space, the farthest point sampling is needed for the foreground point cloud V fg :

[0085] An initial point S1 is randomly selected, and the remaining points are represented as R1 = V fg \S1; at the k-1 step of the iteration process, S k-1 represents the current sampled point set, and the newly selected sampling point S k-1 ′ is determined by finding the point p in the unsampled point set R k-1 that maximizes the distance function D(p, S k-1 ), i.e., Distance function D(p, S k-1 ) = min ||p xyz , s xyz ||2, where p xyz and s xyz represent the three-dimensional coordinates of the unsampled point and the sampled point, respectively; then the set S k = S k-1 ′∪S k-1 , R k = R k-1 \S k is updated and enters the next iteration.

[0086] After iteration, take the set S k As foreground point cloud V fg The result after downsampling is recorded as V sample ;

[0087] S2-3, extract deep learning features:

[0088] The mainstream solution is to use a self-supervised pre-training model, but such a method depends on the performance of the pre-training model. In this embodiment, a hybrid feature representation of traditional geometric features (coordinates, colors) and self-supervised deep learning features is used. The V sample is input into the pre-training model φ (specifically, the PointTransfromer model), to obtain the deep feature F = φ (V sample ), while the point cloud coordinate value xyz is recorded as P, and the color value rgb is recorded as C.

[0089] Step S3 is as follows:

[0090] S3-1, construct a point cloud graph:

[0091] Use the down-sampled point cloud V sample to construct an undirected graph G = {V, E, A}, which uses the k-nearest neighbor method to find the neighboring points of each point and construct edges E with these neighboring points.

[0092] The edge affinity A is composed of the difference values of the depth feature, the coordinate and the color: for the nodes with subscripts i and j in the graph, the cosine similarity is used to calculate the depth feature affinity A (i,j)emb = Cos (F i , F j ), where F is the depth feature of the V sample ; the L2 distance is used to calculate the coordinate affinity A (i,j)pos = -||P i , P j || and the color affinity A (i,j)color = -||C i , C j ||, where P and C are the coordinate information and color information of the V sample ; the final edge affinity A ij = α1A (i,j)emb + α2A (i,j)pos + α3A (i,j)color , where α1, α2 and α3 are hyperparameters.

[0093] S3-2, segment the down-sampled point cloud V sample and assign labels:

[0094] Using the fast multi-cut algorithm RAMA, the minimum segmentation graph is formed by solving the following optimization problem:

[0095]

[0096] where x ij ∈{0,1} represents whether to cut edge (i,j), represents the set of all paths from node v i to node v j , (u,v) represents the two endpoints of an edge in path Path i→j , and traversing all edges in the path; the optimization objective is to minimize the sum of the cut edge weights ∑ (i,j)∈E A ij x ij , with the constraint that if two nodes v i , v j belong to the same subgraph after segmentation, then all edges in the path between them are not cut x uv =0, i.e., to ensure the connectivity of the subgraph; after segmentation into multiple point sets, an initial pseudo-label is assigned to each point set, and the label set is denoted as L sampled .

[0097] Step S4 is specifically as follows:

[0098] The L sampled is restored to the size of the V fg by neighborhood search: first, find the labels of the k closest points in the V fg to each point in the V sample using the k-nearest neighbor method, and then perform majority voting on the k points, with the formula being:

[0099]

[0100] where LU represents the set of candidate labels, l represents a label l selected from the label set LU, N k represents the set of subscripts 1 to k, and I represents 1 if the expression in the parentheses is true, and 0 otherwise; represents the predicted label, l i represents the labels of the k neighboring points; after calculation, the predicted label of each foreground point set is obtained, and the set of the predicted labels is the foreground label; then, the points in the V bg are taken as the background, and the background label is assigned by using the majority voting method; the foreground label and the background label jointly form the complete pseudo-label L of the scene point cloud V full .

[0101] Step S5 is specifically as follows:

[0102] The Mask3D segmentation model is selected for training, and the input is S1 and the collected scene point cloud V full and the complete pseudo label L generated by S4.

[0103] S4-1, construct a loss function:

[0104] For the coordinate center, the training target is to minimize the Euclidean distance between the predicted coordinate center and the real coordinate center, and the loss value is represented as:

[0105] L mean = Euclidean (avg (m), avg (m * ))

[0106] Where m is the real point set, m * is the predicted point set, and avg is the average value.

[0107] For the bounding box, the loss of the bounding box is calculated by the Euclidean distance between the maximum and minimum values of the two point sets, and the loss value is represented as:

[0108] L box = sum (Euclidean (max (m), max (m * )), Euclidean (min (m), min (m * ))

[0109] In addition, there are cross-entropy loss L BCE and Dice loss L dice , represented as:

[0110]

[0111] Where lt i is the real label, is the predicted label, and N is the number of points in the scene point cloud V full .

[0112] The final loss is:

[0113] L = λ1L dice + λ2L BCE + λ3L mean + λ4L box

[0114] Where λ1, λ2, λ3, λ4 are weight parameters.

[0115] In order to be able to balance local details and overall structure, this embodiment adopts two steps of over-segmentation training and under-segmentation optimization:

[0116] S4-2, Over-segmentation training: By adjusting the hyperparameters of the multi-cut algorithm, the segmentation result is more fine-grained. The segmentation result will present more point sets and more different manifestations. The manifestation is that the object will be split into multiple parts (for example, a chair may be divided into a backrest, a cushion, etc.). This operation makes the model learn local features, such as the geometric structure and color consistency of the object surface. At this time, the model may predict a large number of small objects.

[0117] S4-3, Under-segmentation optimization: In S4-2, each object in the scene is finely segmented, but the fine-grained segmentation causes two parts that should belong to the same object to be unable to be associated together. Therefore, under-segmentation optimization is adopted to fine-tune with a coarser pseudo label (which may merge adjacent objects) to prompt the model to learn the global structure and merge multiple small point sets belonging to the same object. The pseudo label is trained for a small number of iterations, and the bounding box loss and mean loss are still used for optimization. After several iterations, the integrity of the instance can be greatly increased.

[0118] Step S6 is as follows:

[0119] Prepare 3 depth cameras in 3 directions and deploy them to different positions in the scene; after the device is deployed, record the coordinates [x0, y0, z0] of each depth camera; use the deployed depth camera to take pictures and obtain multiple depth images I depth ; The depth image contains RGB-D information, which means that a pixel in the picture has RGB color information and D depth information; apply Gaussian filtering to each depth image to remove noise;

[0120] Then calculate the position of the point cloud through the depth information and the position of the picture, and save the mapping relationship between the pixel and the corresponding point cloud; first calculate the relative coordinates x c ,y c ,z c :

[0121]

[0122] wherein u, v represent any coordinate point in the image, u0, v0 represent the center of the image, d is the depth information, f is the focal length of the camera, x0, y0, z0 is the coordinate of the camera, dx represents the physical size of the pixel in the x-axis direction, and dy represents the physical size of the pixel in the y-axis direction;

[0123] Then calculate the unified world coordinates according to the relative coordinates: [x w ,y w ,z w ] T = R·[x c ,y c ,z c] T +[x0,y0,z0], where R is the rotation matrix;

[0124] For point cloud color information, use the average value of images at various depths. The calculation shows that n is the total number of depth images, and Ci is the RGB value of the i-th image;

[0125] Finally, the scene point cloud V of the target scene to be segmented, including coordinate and color information, is obtained. target , put V target Input the trained point cloud instance segmentation model to obtain the instance segmentation result of the point cloud.

[0126] Accordingly, the present invention also provides an unsupervised point cloud segmentation system based on multi-feature fusion and graph optimization, such as... Figure 2 As shown, it includes:

[0127] Point cloud information extraction unit 1 is used to extract scene point cloud V containing coordinate and color information for any data item in the 3D indoor scene dataset as training data. full The scene point cloud V full Divided into foreground viewpoints and cloud V fg and background point cloud V bg Then, for the previous scenic spot, Cloud V fg Perform farthest point sampling to obtain downsampled point cloud V sample Then downsample the point cloud V sample Deep learning features F and V are extracted from the pre-trained model φ. sample The coordinate information is denoted as P, and the color information is denoted as C;

[0128] Sampling label generation unit 2 is used to utilize the downsampled point cloud V sample Construct an undirected graph G = {V, E, A} containing vertices V, edges E, and edge affinities A. Then, use a fast multi-cut algorithm to solve the minimum partitioning problem, forming V. sampled The segmentation result contains multiple point sets. An initial pseudo-label is then assigned to each point set. This label set is denoted as L. sampled ;

[0129] Complete tag generation unit 3, used to generate the L sampled Upsampled to the foreground cloud V fg Obtain the foreground label, which is the background point cloud V. bg Assign background labels; foreground labels and background labels together form the scene point cloud V. full The complete pseudo-label L; the model used to generate the complete pseudo-label is denoted as the point cloud segmentation model;

[0130] The model training unit 4 is configured to input the scene point cloud V full The complete pseudo label L is used to train the point cloud segmentation model by using a Mask3D segmentation model and constructing a weak supervision loss function composed of a coordinate center loss, a bounding box loss, a cross entropy loss and a Dice loss, through two steps of over-segmentation training and under-segmentation optimization.

[0131] The deployment application unit 5 is configured to, for a target scene to be segmented, use a depth camera deployed at different positions in the target scene to capture a depth image including color information and depth information, then calculate point cloud coordinates by using the depth information and calculate point cloud colors by using the color information, and obtain a scene point cloud V target containing coordinate information and color information.

[0132] Therefore, the novel graph construction and distribution method is used to perform unsupervised learning of the point cloud, and the problems of shortage of point cloud data sets and difficulty in labeling are solved. Compared with other unsupervised learning methods, the traditional point features and pre-training features are considered, the performance dependence on the pre-training model is reduced, and the reliability of the method is improved. Meanwhile, the foreground separation method is innovatively used according to the characteristics of the indoor scene, the foreground object distribution is diverse, and the structure is complex, so that the model can be optimized for the foreground effect, and better segmentation effect is achieved.

[0133] The above describes a multi-feature fusion and graph optimization unsupervised point cloud segmentation method and system provided by the embodiment of the present application in detail, and the principle and implementation mode of the present application are described by using specific examples. The above embodiment description is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation mode and application range will be changed according to the idea of the present application. Therefore, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An unsupervised point cloud segmentation method based on multi-feature fusion and graph optimization, characterized in that, The method comprises: Using a 3D indoor scene dataset as training data, for any data item in the training data, extract the scene point cloud V full containing coordinate information and color information thereof The scene point cloud V full is segmented into a foreground point cloud V fg and a background point cloud V bg , then the foreground point cloud V fg is subjected to farthest point sampling to obtain a down-sampled point cloud V sample , and then the down-sampled point cloud V sample is input into a pre-trained model φ to extract a deep learning feature F, and the coordinate information of V sample is recorded as P and the color information is recorded as C. The down-sampled point cloud V sample An undirected graph G = {V, E, A} is constructed, containing points V, edges E and edge affinity A, and then a fast multi-cut algorithm is used to solve the minimum cut problem, forming V sample The segmentation result of V sampled , which contains multiple point sets, and then an initial pseudo label is assigned to each point set, and the label set is denoted as L The L sampled is up-sampled to the foreground point cloud V fg The foreground label is obtained, and the background point cloud V bg is assigned a background label; the foreground label and the background label jointly form a complete pseudo label L of the scene point cloud V full ; a model for generating the complete pseudo label is denoted as a point cloud segmentation model; inputting the scene point cloud V full , the complete pseudo label L, using a Mask3D segmentation model, and constructing a weakly supervised loss function composed of a coordinate center loss, a bounding box loss, a cross entropy loss, and a Dice loss, training the point cloud segmentation model through two steps of over-segmentation training and under-segmentation optimization, and forming a trained point cloud instance segmentation model; For a target scene to be segmented, a user uses a depth camera deployed at different positions in the target scene to take pictures, obtaining a depth image including color information and depth information, then calculates point cloud coordinates using the depth information, calculates point cloud colors using the color information, and obtains a scene point cloud V target containing coordinate information and color information. Then, inputting the trained point cloud instance segmentation model can obtain an instance segmentation result of the point cloud.

2. The unsupervised point cloud segmentation method of claim 1, wherein, The 3D indoor scene dataset comprises ScanNet and S3DIS.

3. The unsupervised point cloud segmentation method of claim 1, wherein, The scene point cloud V full is divided into a foreground point cloud V fg and a background point cloud V bg , then the foreground point cloud V fg is subjected to farthest point sampling to obtain a down-sampled point cloud V sample , and then the down-sampled point cloud V sample is input into a pre-trained model φ to extract a deep learning feature F, specifically: One of them, the scene point cloud V full background separation is performed: First, the bounding box is determined by the extreme values in each direction, and the surface normal vector of the bounding box is taken as the normal vector of the candidate plane; the candidate plane is represented as where i is the subscript of the surface, q is the column vector [x, y, z] T , n i is the normal vector of each surface, and d i is the intercept; For each candidate plane, the intersection over union IOU of its projection with each bounding box surface is calculated, where Area represents the projected area, H i is the in-plane point set, is the orthogonal projection function, B i is the bounding box surface; when the intersection-over-union is greater than a pre-set threshold, it is removed as the background in the scene, and the foreground point cloud is retained; at this point, the scene point cloud V full is divided into foreground point cloud V fg and background point cloud V bg ; Secondly, the foreground point cloud V fg furthest point sampling is performed: Randomly select initial point S1, and the rest of the points are denoted as R1=V fg S1; at the k-1th step of the iteration process, S k-1 denotes the current sampled point set, and the newly selected sampling point S k-1 ′ is determined by finding the point p in the unsampled point set R k-1 which can maximize the distance function D(p, S k-1 ), that is, The distance function D(p, S k-1 ) = min ||p xyz -s xyz ||2, where p xyz and s xyz represent the three-dimensional coordinates of the unsampled points and the sampled points, respectively; then update the set S k = S k-1 ′∪S k-1 , R k = R k-1 \S k enter the next iteration step; After the iteration is completed, take the set S k As the foreground point cloud V fg The result after downsampling is denoted as V sample ; Thirdly, deep learning features are extracted: The V sample input pre-training model φ, to obtain a deep feature F = φ(V sample ).

4. The unsupervised point cloud segmentation method of claim 1, wherein, The utilization of the down-sampled point cloud V sample An undirected graph G = {V, E, A} is constructed, which contains points V, edges E and edge affinity A, and then a fast multi-cut algorithm is used to solve the minimum segmentation problem, forming V sample The segmentation result of V sampled , and an initial pseudo-label is assigned to each point set, and the label set is denoted as L sampled , specifically: Firstly, a point cloud graph is constructed: The down-sampled point cloud V is used sample The k-nearest neighbor method is used when constructing the undirected graph G = {V, E, A}, finding the neighboring points of each point, and constructing edges E with these neighboring points; The edge affinity A is composed of the difference values of the depth feature, the coordinate and the color: for the node with subscript i, j in the graph, the cosine similarity is used to calculate the depth feature affinity A (i,j)emb = Cos(F i , F j ), wherein F is the depth feature of the V sample ; the L2 distance is used to calculate the coordinate affinity A (i,j)pos = -||P i , P j || and the color affinity A (i,j)color = -||C i , C j ||, wherein P and C are the coordinate information and the color information of the V sample ; the final edge affinity A ij = α1A (i,j)emb + α2A (i，j)pos + α3A (i，j)color , wherein α1, α2 and α3 are hyperparameters; Second, the down-sampled point cloud V sample Segmentation and label assignment: A minimum segmentation graph is formed by solving the following optimization problem using a fast multi-cut algorithm RAMA: where x ij ∈ {0, 1} indicates whether the cut edge (i, j) is cut, denotes the node v i to the node v j the set of all paths, (u, v) denotes the path Path i→j two endpoints of a segment of an edge in the path, traverse all edges in the path; the optimization objective is to minimize the sum of the cut edge weights (i,j)∈ E A ij x ij , the constraint condition is that if two nodes v i , v j belong to the same subgraph after partitioning, then all edges on the path of the two nodes are not cut x uv = 0, that is, to ensure the connectivity of the subgraph; after partitioning into multiple point sets, an initial pseudo label is assigned to each point set, and the label set is denoted as L sampled .

5. The unsupervised point cloud segmentation method of claim 1, wherein, The L sampled upsampling to the foreground point cloud V fg Get foreground label, for the background point cloud V bg Assign background label; The foreground label and the background label jointly form the complete pseudo label L of the scene point cloud V full , specifically: The L sampled is recovered to the V fg Scale: First find the label of the k nearest neighbors of each point in the V fg in the V sample , then majority vote on the k points, formula: where LU denotes a set of candidate labels, l denotes a label l selected from the set of labels LU, N k denotes a set of indices 1 to k, I denotes 1 if the expression inside the brackets is true, otherwise 0; denotes a predicted label, l i denotes the labels of the k nearest neighbors; After the calculation, the predicted label of each foreground point set is obtained, and the set is a foreground label; then the points in the V bg are taken as background, and a background label is obtained by using the majority voting method; the foreground label and the background label jointly form the complete pseudo label L of the scene point cloud V full .

6. The unsupervised point cloud segmentation method of claim 1, wherein, The weakly supervised loss function composed of coordinate center loss, bounding box loss, cross-entropy loss and Dice loss is constructed, and specifically: For the coordinate center, the training target is to minimize the Euclidean distance between the predicted coordinate center and the real coordinate center, and the loss value is represented as: L mean = Euclidean(avg(m), avg(m * )) where m is the set of real points, m * is the set of predicted points, avg is the average. For the bounding box, the loss of the bounding box is calculated by the Euclidean distance of the maximum and minimum values of the two point sets, and the loss value is represented as: L box = sum(Euclidean(max(m),max(m * )), Euclidean(min(m),min(m * )) In addition, there is a cross-entropy loss L BCE and Dice loss L dice , denoted as: wherein lt i is a true label, is a predicted label, N is the number of points of the scene point cloud V full . The final loss is: L = λ1L dice + λ2L BCE + λ3L mean + λ4L box Where λ1, λ2, λ3, λ4 are weight parameters.

7. The unsupervised point cloud segmentation method of claim 1, wherein, The user uses a depth camera arranged at different positions in the target scene to take pictures to obtain a depth image including color information and depth information, calculates point cloud coordinates using the depth information, calculates point cloud colors using the color information, and obtains a scene point cloud V target The trained point cloud instance segmentation model is inputted to obtain an instance segmentation result of the point cloud, specifically: Prepare three depth cameras in three directions, and deploy them to different positions in the scene; after the deployment of the equipment is completed, record the coordinates [x0, y0, z0] of each depth camera; use the deployed depth camera to take pictures and obtain multiple depth images I depth ; the depth image contains RGB-D information, and the RGB-D information refers to the RGB color information and D depth information of a pixel in the picture; Gaussian filtering is applied to each depth image to remove noise; After that, the position of the point cloud is calculated by the depth information and the photo position, and the mapping relationship between the pixel and the corresponding point cloud is saved; first, the relative coordinates x c ,y c ,z c relative to the camera are calculated. Where u, v represent any coordinate points in the image, u0, v0 represent the image center, d is the depth information, f is the camera focal length, x0, y0, z0 are the coordinates of the camera, dx represents the physical size of the pixel in the x-axis direction, and dy represents the physical size of the pixel in the y-axis direction. From the relative coordinates, the unified world coordinates are calculated: [x w ,y w ,z w ] T = R - [x c ,y c ,z c ] T + [x0, y0, z0], where R is the rotation matrix; For point cloud color information, use the average of each depth picture is computed, where n is the total number of depth pictures and Ci is the RGB value of the i-th picture; Finally, the scene point cloud V of the target scene to be segmented including coordinate information and color information is obtained target V target The trained point cloud instance segmentation model is input to obtain the instance segmentation result of the point cloud.

8. An unsupervised point cloud segmentation system with multi-feature fusion and graph optimization, characterized in that, The system comprises: The point cloud information extraction unit is used to extract scene point cloud V containing coordinate and color information for any data item in the 3D indoor scene dataset as training data. full The scene point cloud V full Divided into foreground viewpoints and cloud V fg and background point cloud V bg Then, for the previous scenic spot, Cloud V fg Perform farthest point sampling to obtain downsampled point cloud V sample Then downsample the point cloud V sample Deep learning features F and V are extracted from the pre-trained model φ. sample The coordinate information is denoted as P, and the color information is denoted as C; A sampling label generation unit is configured to generate a down-sampled point cloud V sample An undirected graph G = {V, E, A} containing points V, edges E, and edge affinities A is constructed, and then a fast multi-cut algorithm is used to solve a minimum cut problem to form V sample segmentation results, which contain a plurality of point sets, and then an initial pseudo label is assigned to each point set, and a label set is denoted as L sampled ; a complete label generation unit configured to generate a complete label L for the foreground point cloud V sampled up-sampling the foreground point cloud V fg to obtain a foreground label for the foreground point cloud V bg assigning a background label; the foreground label and the background label together form a complete pseudo label L for the scene point cloud V full ; a model used for generating the complete pseudo label is denoted as a point cloud segmentation model; A model training unit is configured to input the scene point cloud V full The complete pseudo label L is used to train the point cloud segmentation model by using a Mask3D segmentation model and constructing a weak supervision loss function composed of a coordinate center loss, a bounding box loss, a cross entropy loss and a Dice loss. The point cloud segmentation model is trained through two steps of over-segmentation training and under-segmentation optimization, and a trained point cloud instance segmentation model is formed. The application unit is deployed for a target scene to be segmented, a user uses a depth camera deployed at different positions in the target scene to take pictures, and a depth image including color information and depth information is obtained. Then, the point cloud coordinates are calculated using the depth information, and the point cloud color is calculated using the color information, so as to obtain a scene point cloud V target which contains coordinate information and color information. Then, the trained point cloud instance segmentation model is input, and the instance segmentation result of the point cloud is obtained. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 7.

Images