An example relationship-driven large-scale garden scene point cloud segmentation method

Abstract

The application discloses a kind of instance relationship-driven large-scale garden scene point cloud segmentation methods, comprising: first using the method of downsampling and clustering obtains the instance object annotation data of garden scene point cloud, then using annotation data trains instance segmentation network, adopts the instance object in point cloud that is segmented out using trained instance segmentation network, then using garden point cloud and instance object input to instance relationship graph generator generates the instance relationship graph of garden scene, again through graph convolution generates the instance relationship graph feature of each garden scene, finally the instance relationship graph feature of garden scene and point cloud connection input to trained point cloud segmentation network obtains the point-by-point segmentation data of large-scale garden scene.The application can cope with large-scale garden scene point cloud, can accurately obtain the context information in garden point cloud scene, provide sufficient prior knowledge for the point cloud segmentation of garden scene, so as to improve the segmentation efficiency and accuracy of garden point cloud.

Description

TECHNICAL FIELD

[0001] The present application relates to a garden scene point cloud segmentation method, in particular to an instance relationship driven large-scale garden scene point cloud segmentation method. BACKGROUND

[0002] There are rich and varied landscapes in the garden scene, and it is an important link to study garden data to segment these different landscapes from the scene by using computer vision means. It helps to deepen the study of garden data, realize the analysis of the content structure of the garden scene, and further protect these intangible cultural heritage. Due to the in-depth study of the garden scene in recent years, researchers can obtain larger and larger garden three-dimensional data. The main form of these large-scale garden three-dimensional data is mainly point cloud data collected by laser or multi-view reconstruction. Compared with the relatively mature indoor and outdoor public point cloud data sets, the garden data has the characteristics of large density and complex structure, which brings great difficulty to the processing of point cloud and becomes a big problem of point cloud segmentation.

[0003] With the gradual rise of deep learning, complex point cloud segmentation problems are given to more and more advanced deep neural networks to solve, and point cloud segmentation methods using deep neural networks as tools gradually become the mainstream of point cloud segmentation. Literature 1: Qi C R, Su H, Mo K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 652-660, through MLP (Multilayer Perceptron, Multilayer Perceptron) Map all points in the point set to a high-dimensional feature space, and then aggregate features through maximum pooling to obtain the global information of the point cloud, which first introduced deep learning into point cloud processing. On this basis, literature 2: Qi C R, Yi L, Su H, et al. Pointnet ++: Deep hierarchical feature learning on point sets in a metric space [J]. Advances in neural information processing systems, 2017, 30. In the local neighborhood, the network is used to extract local features, and then through multiple stepwise downsampling, local to global features of different scales and levels are obtained, and finally a decoder is used to aggregate features, which has achieved good results. In recent years, literature 3: Zhao H, Jiang L, Jia J, et al. Point transformer [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 16259-16268. A network module based on self-attention mechanism is introduced for point cloud segmentation, which can better capture the long-distance context in the point cloud. These point cloud processing methods, although have achieved good results on public datasets, are all point cloud segmentation networks acting on indoor data with simple context relationships between scenes and objects, and single object forms. However, for outdoor datasets, SLAM (Simultaneous Localization And Mapping, Simultaneous Localization And Mapping) datasets with relatively simple data distribution are mostly used.The effects of these methods on the garden dataset are not satisfactory. The irregular rockery and green plants in the garden data can interfere with the extraction of local features in these methods and interfere with the transmission of global and long-distance information.

[0004] In view of the particularity of the garden data, document 4: Ai Qiao, Zhou Weishi, Yao Yang, Dong Lili, Liu Hua, Gu Han. A garden image quantification method based on semantic segmentation [P]. Chongqing: CN114202546A, 2022-03-18. proposed a semantic segmentation method for garden data set, which converts the garden image set into a form conducive to semantic segmentation, and then quantizes it. This method mainly processes garden data on images and cannot be used for garden three-dimensional point cloud processing. Document 5: Meng Yanmei, Li Ke, Han Bing, Zhang Jinlai, Wei Jin, Liu Hui. Garden map construction method based on multi-scale distance graph and point cloud semantic segmentation [P]. Guangxi Zhuang Autonomous Region: CN114926637A, 2022-08-19. proposed a method combining multi-scale distance graph construction and point cloud segmentation for garden data to solve the problem of unclear semantics in point cloud. However, this method only uses distance graph to project point cloud, and does not solve the problem of unclear context in garden point cloud. Moreover, the point cloud segmentation is mainly used for map construction, and a mature semantic segmentation network is used, so there is no improvement in point cloud segmentation.

[0005] Summarizing the characteristics of existing research and application of garden three-dimensional point cloud segmentation methods, the following shortcomings exist in the current three-dimensional point cloud segmentation applied to garden data:

[0006] 1. Insufficient training data. Existing point cloud segmentation methods are mostly trained on public data sets, and there is no annotated garden data for existing methods to train, which makes existing methods unable to fully experiment on garden data and not well adapted to garden data sets.

[0007] 2. Existing methods are not optimized for garden data. Garden point cloud data has a different data distribution than conventional indoor and outdoor data sets. At the same time, the context clues and relationships in garden data are different from public data sets, and using methods from public data sets will cause errors in the extraction and transmission of local features and global information. SUMMARY

[0008] The technical problem to be solved by the present application is to provide an instance relationship driven large-scale garden scene point cloud segmentation method to overcome the shortcomings of the prior art.

[0009] To solve the above technical problems, the present application discloses an instance relationship driven large-scale garden scene point cloud segmentation method, comprising the following steps:

[0010] Step 1, instantiate and label the garden scene point cloud: For the garden scene point cloud data, firstly, downsampling is used to reduce the data density, then Euclidean clustering is used to cluster the instance objects in the point cloud, and finally, different instance labels are assigned according to the point cloud category.

[0011] Furthermore, the instantiation and annotation of the point cloud of the garden scene includes the following steps:

[0012] Step 1-1, for the garden scene point cloud P∈R to be segmented N×3 , where R N×3 Represents N three-dimensional coordinates belonging to the real number domain, uniformly downsampled to the downsampled point cloud P using the farthest point sampling method. s The specific method is as follows:

[0013]

[0014] Where d represents the point set, and P is the downsampled point cloud. s It consists of a point set d, where S represents the sample point set, which refers to the original points in the garden scene point cloud, p represents the point farthest from the query point among the sample points, q represents the query point among the sample points, A represents the downsampled region in the garden scene point cloud, and N ds This represents the total number of points in the downsampled region;

[0015] Steps 1-2, for the downsampled point cloud P s Using Euclidean clustering, the point cloud P is... s The points are clustered into different clusters, and a preset Euclidean distance is used as the clustering threshold D. The Euclidean distance between points in the point cloud is calculated as follows:

[0016]

[0017] Where, x i ,x j Representative point cloud P s Two points in the middle, x ik ,x jk The coordinates of the two points mentioned above are represented by d, where d represents the total dimension of the point cloud and k represents the corresponding dimension.

[0018] Furthermore, the Euclidean clustering method specifically includes:

[0019] First, calculate the point cloud P. s The Euclidean distance between two points in space is calculated, and then compared with a set threshold. Points with a distance less than the threshold are clustered into a cluster, while points with a distance greater than the threshold are clustered into another cluster.

[0020] Step 1-3, first, the number of points in the cluster obtained in step 1-2 is between N min and N max , according to shape division, it is judged whether there is a connection relationship between the clusters of the same shape and the clusters, and accordingly it is divided into different instance objects; wherein, N min is a preset minimum number of clustering points, N max is a preset maximum number of clustering points;

[0021] Then in the cluster clustered in step 1-2, the points greater than the minimum number of points in the cluster are divided into different instance objects according to the original point cloud annotation; the points less than the minimum number of points in the cluster , if they can be spliced into instance objects through the connection relationship, they are spliced into instance objects, otherwise they are marked as non-instance objects, to obtain instance object annotation data.

[0022] Step 2, instance segmentation of garden scene point cloud: using the garden scene point cloud data annotated with instances to train an instance segmentation network, and then using the instance segmentation network to extract instances from the garden scene point cloud;

[0023] Further, the instance segmentation of the garden scene point cloud comprises the following steps:

[0024] Step 2-1, using the instance annotation data in step 1 to train the point cloud feature extraction network and the instance segmentation network, the instance segmentation network is a three-dimensional point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution, using the cross-entropy loss L cls , L1 and gIoU combined loss L box , dice and BCE combined loss L mask and mask score L MS loss, the total loss L total is obtained by combination, the total loss L total is used as a loss function to train the point cloud feature extraction network and the instance segmentation network;

[0025] Further, the total loss L total is expressed as follows:

[0026] L total =L cls +λ box L box +λ mask L mask +λ ms L MS

[0027] Wherein, L total represents the total loss, λ box , λ mask and λms respectively represent the weights of loss L box , L mask and L MS , L cls represents the cross-entropy loss, L box represents the L1 and gIoU combined loss, L mask represents the dice and BCE combined loss, L MS represents the mask score loss.

[0028] Step 2-2, input the garden scene point cloud into the trained point cloud feature extraction network to obtain the point cloud feature F∈R N×D , where R represents the real number field, N represents the number of points, and D represents the output dimension;

[0029] Step 2-3, input the point cloud feature F into the trained instance segmentation network to obtain the segmented instance object.

[0030] Further, the segmented instance object includes the following steps:

[0031] Step 2-3-1, input the point cloud feature F into the trained instance segmentation network based on the sampled instance encoder and point predictor to extract features, the instance encoder outputs the kernel prediction W∈R K×H' and the box prediction B∈R K×6' , the point predictor outputs the mask feature F mask ∈R N×H and the box feature F box ∈R N×6 .

[0032] Step 2-3-2, input the instance encoder output features W, B and the point predictor output features F mask ,F box into the box-aware dynamic convolution to output the instance segmentation result.

[0033] Further, the processing process of the box-aware dynamic convolution is:

[0034]

[0035] wherein, represents the mask of the finally generated instance, i.e. the instance segmentation result, Sigmoid represents the sigmoid function, Conv represents the convolution operation, and F mask . represents different features, w (k) represents the instance kernel.

[0036] Step 3, generating an instance relation graph feature of the garden scene point cloud: according to the obtained instances in the garden scene point cloud, combining the garden scene point cloud, generating an instance relation graph in the garden scene point cloud, and obtaining a relation graph feature of the garden scene point cloud through graph convolution;

[0037] Further, the instance relation graph feature of the garden scene point cloud comprises the following steps:

[0038] Step 3-1, inputting N M instances segmented by the instance segmentation network into a linear layer in the instance relation graph generator to generate a node feature set in the garden relation graph Where C m represents the number of feature channels;

[0039] Step 3-2, inputting the garden scene point cloud and the above instance into a linear layer in the relation graph generator respectively to obtain the weight of N×N edges, combining the graph node feature set M g to obtain a garden instance relation graph

[0040] Further, the garden instance relation graph is represented as:

[0041]

[0042] Wherein, represents the garden instance relation graph, represents the instance node, and ε represents the edge, M g represents the node feature set, and A represents the adjacency matrix.

[0043] Step 3-3, processing the garden instance relation graph using a graph convolution operation to obtain a garden instance relation graph feature Specifically represented as:

[0044]

[0045] Wherein, represents the feature after graph convolution, I represents the unit matrix, and W7 represents the learnable weight matrix.

[0046] Step 4, point cloud segmentation of the garden scene point cloud: connecting the garden scene point cloud and the relation graph feature to input into the trained point cloud segmentation network to obtain the point-by-point class information of the garden scene point cloud, and completing the instance relation driven large-scale garden scene point cloud segmentation.

[0047] Further, the point cloud segmentation of the garden scene point cloud comprises the following steps:

[0048] Step 4-1, training a point cloud segmentation network;

[0049] Step 4-2, the garden scene point cloud P and the garden instance relationship graph feature Connect and input to the trained point cloud segmentation network to obtain a point-by-point class label, and complete the instance relationship driven garden scene point cloud segmentation.

[0050] Advantages:

[0051] 1. The garden point cloud data labeled on a large scale is used as training data in the application, which avoids misleading the network by other types of data priori, and fully utilizes the prior knowledge in the garden data to train the network, which is beneficial to the adaptation of the method to the garden data domain.

[0052] 2. According to the characteristics of complex point cloud structure and unclear context clues in the garden point cloud, the application adopts instance segmentation to divide the point cloud into clear semantic regions, and adopts an instance relationship graph to extract the context clues in the point cloud. The garden semantic region composed of multiple different instances clearly analyzes the complex structure in the point cloud, and simplifies the context relationship in the garden point cloud data. The instance relationship graph composed of these instances more clearly determines the context relationship and the connection between objects in the garden data, and the instance relationship features obtained therefrom can guide the network to extract the local structural features and global long-distance information in the point cloud data, thereby guiding the point cloud segmentation network to accurately segment the garden point cloud data and segment it into different categories point by point.

[0053] 3. The application is applicable to the processing of large-scale and different density garden point cloud data, and different instance relationship features can be obtained according to different types of garden point cloud, which makes the application method applicable to different types of garden point cloud data.

[0054] 4. Compared with the existing three-dimensional point cloud segmentation method, the application is optimized according to the characteristics of the garden point cloud, and is superior to most point cloud segmentation methods in the segmentation of the garden point cloud. BRIEF DESCRIPTION OF DRAWINGS

[0055] The above and / or other aspects of the application will become more apparent by describing in detail the application with reference to the accompanying drawings, in conjunction with the specific embodiments.

[0056] Figure 1 is a processing flow schematic diagram of the application.

[0057] Figure 2 is a visualization result of the input garden point cloud.

[0058] Figure 3 is a garden instance object labeling visualization result.

[0059] Figure 4 is a garden instance object segmentation visualization result.

[0060] Figure 5 is a garden instance relationship graph visualization result.

[0061] Figure 6 is a garden scene point cloud segmentation result visualization result. DETAILED DESCRIPTION

[0062] The application discloses a large-scale garden scene point cloud segmentation method driven by instance relationship, comprising the following steps:

[0063] Step 1, select part of the garden point cloud, and perform instance labeling on the garden point cloud;

[0064] Step 2, training a three-dimensional point cloud feature extraction network and an instance segmentation network using the labeled data, and performing instance segmentation using the network;

[0065] Step 3, inputting the instance segmentation result and the three-dimensional point cloud into an instance relationship generation network, constructing an instance relationship graph, and inputting the instance relationship graph into a graph convolution network to obtain a garden instance relationship graph feature;

[0066] Step 4, training a point cloud relationship segmentation network using the garden instance relationship graph feature together with large-scale garden point cloud data, and segmenting the garden point cloud data using the network.

[0067] Step 1 comprises the following steps:

[0068] Step 1-1, randomly selecting part of the point cloud from the large-scale garden point cloud data, and uniformly downsampling the point cloud using a uniform downsampling method.

[0069] Step 1-2, clustering the points in the point cloud into different clusters in the form of Euclidean distance clustering.

[0070] Step 1-3, according to the positions of different point clusters, combining the original point cloud label, and giving different instance labels to different clusters.

[0071] In step 1-1, the furthest point sampling method is used to uniformly downsample the point cloud, and the formula is as follows:

[0072]

[0073] Wherein d represents a point set, S represents a sample set, p represents a furthest point, q represents a query point, A represents a given region, and N represents the total number of points.

[0074] In step 1-2, a certain Euclidean distance is used as the clustering threshold to cluster the point cloud into different instances, and the formula for calculating the Euclidean distance is as follows:

[0075]

[0076] where x i ,x j represents two points, x ik ,x jk represent the coordinates of two points, d represents the total dimension, and k represents the corresponding dimension.

[0077] Step 2 includes the following steps:

[0078] Step 2-1, according to the labeled data, train literature 2: Qi C R, Yi L, Su H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space [J]. Advances in neural information processing systems, 2017, 30. Point cloud feature extraction network (PointNet++) and literature 6: Ngo T D, Hua B S, Nguyen K. ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 13550-13559. Instance-aware sampling and box-aware dynamic convolution of three-dimensional point cloud instance segmentation network (ISBNet, Instance-aware Sampling and Box-aware Network).

[0079] Step 2-2, input the point cloud into the point cloud feature extraction network to extract features.

[0080] Step 2-3, input the point cloud features into the sampling-based instance-wise encoder and point-wise predictor in the instance segmentation network to extract features.

[0081] Step 2-4, input the output features of the point-wise predictor and the output features of the instance encoder into the box-aware dynamic convolution, and output the instance segmentation result.

[0082] In step 2-1, the loss function for training the instance segmentation network is defined as follows:

[0083] L total =L cls +λ box L box +λ mask L mask +λ ms L MS

[0084] where L total represents the total loss, λ represents the weight of each loss, L cls represents the cross-entropy loss, L box represents the L1 and gIoU combined loss, L mask represents the dice and BCE combined loss, L MS represents the mask score loss.

[0085] In step 2-2, the point cloud feature extraction network is used to extract the point cloud features, and the output of the feature extraction network is:

[0086] F∈R N×D

[0087] where F represents the output feature, R represents the real number field, N represents the number of points, and D represents the output dimension.

[0088] In step 2-3, the instance-aware sampling and box-aware dynamic convolution based on the sampling instance encoder in document 6: Ngo T D, Hua B S, Nguyen K. ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 13550-13559. is used, and the formula is:

[0089]

[0090] where P' represents the sampling point, K' represents the selected candidate point, represents the probability of the point as background, and τ is the threshold value of the hyperparameter.

[0091] In step 2-4, the processing process of the box-aware dynamic convolution can be represented by the formula:

[0092]

[0093] wherein, represents the mask of the last generated instance, Sigmoid represents the sigmoid function, Conv represents the convolution operation, F mask ; represents different features, w (k) represents the instance kernel.

[0094] Step 3 includes the following steps:

[0095] Step 3-1, input the segmented instance into the instance relation graph generator composed of multiple linear layers to generate a node feature set in the garden relation graph wherein C m represents the number of feature channels.

[0096] Step 3-2, input the three-dimensional point cloud and the instance into the instance relation graph generator to generate an edge in the garden relation graph, and combine the graph node feature set to generate a garden instance relation graph.

[0097] Step 3-3, use graph convolution to process the garden instance relation graph to generate garden instance relation graph features.

[0098] In step 3-2, the generated garden instance relation graph can be represented as:

[0099]

[0100] wherein, represents the garden instance relation graph, represents the instance node, and ε represents the edge. g represents the feature set of the node, and A represents the adjacency matrix.

[0101] In step 3-3, the garden instance relation graph is processed using graph convolution, which can be represented by the formula

[0102]

[0103] wherein, represents the feature after graph convolution, I represents the unit matrix, and W7 represents the learnable weight matrix.

[0104] Step 4 includes the following steps:

[0105] Step 4-1, using the garden relationship graph features and large-scale garden point cloud data to train the point cloud segmentation network in the prior art (for example: the point cloud segmentation network used in document 7: Thomas H, Qi C R, Deschaud J E, et al. Kpconv: Flexible and deformable convolution for point clouds [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6411-6420, or other point cloud segmentation networks).

[0106] Step 4-2, input the garden instance relationship graph features and large-scale garden point cloud data of the corresponding scene to the trained point cloud segmentation network, and generate point-by-point class labels after segmentation.

[0107] Embodiment:

[0108] As shown in Figure 1 An example relationship-driven large-scale garden scene point cloud segmentation method disclosed by the present application specifically includes the following steps:

[0109] Step 1, instance annotation of large-scale garden point cloud: for large-scale garden point cloud data, the data density is reduced by using the furthest point uniform downsampling method, the instance objects in the point cloud are clustered by using the Euclidean clustering method, and different instances are annotated according to the point cloud categories.

[0110] Step 2, instance segmentation of garden point cloud: using the annotated garden data to train the instance segmentation network, and using the network to extract instances of the garden scene point cloud.

[0111] Step 3, generation of garden point cloud instance relationship graph features: according to the obtained instances in the garden point cloud, combining the point cloud, using the instance relationship graph generator to generate the instance relationship graph in the garden point cloud, and obtaining the relationship graph features of the garden point cloud through graph convolution.

[0112] Step 4, point cloud segmentation of large-scale garden point cloud: connecting the large-scale garden scene point cloud and the relationship graph features to input into the trained point cloud segmentation network, and obtaining the point-by-point class information of the garden point cloud.

[0113] The main process of each step is introduced below: Step 1, instance annotation of large-scale garden point cloud, which specifically includes the following process:

[0114] Step 1.1, for an input garden scene point cloud P∈R N×3 As shown in Figure 2A garden scene point cloud is shown, and the point cloud is uniformly down-sampled by using the farthest point sampling method to reduce the point cloud density. The main calculation method is as follows:

[0115]

[0116] wherein d represents a point set, S represents a sample set, p represents a farthest point, q represents a query point, A represents a given region, and N represents the total number of points. In the actual implementation process, A is the overall region of the point cloud, and the obtained down-sampled point cloud is P s .

[0117] Step 1.2, the down-sampled point cloud P s is clustered into different clusters by using the Euclidean clustering method, and the calculation formula of the Euclidean distance is as follows:

[0118]

[0119] wherein x i ,x j represent two points, x ik ,x jk represent the coordinates of the two points, d represents the total dimension, and k represents the corresponding dimension. In the actual implementation process, d = 3, x ik ,x jk are three groups of coordinate points, and D = 0.1.

[0120] Step 1.3, according to the minimum number of points N min and the maximum number of points N max , the clusters are divided into different instances, and then according to the existing point cloud label and the minimum number of points in the cluster , the clusters are divided into different instance objects, as shown in Figure 3 each three-dimensional frame is an instance clustered. In the actual implementation process, N min = 100, N max = 10000000,

[0121] Step 2, instance segmentation of the garden point cloud, the specific process is as follows:

[0122] Step 2.1, use the instance annotation data in step 1 to train the point cloud feature extraction network and the three-dimensional point cloud instance segmentation network. The feature extraction network uses PointNet++ in document 2: Qi C R, Yi L, Su H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space [J]. Advances in neural information processing systems, 2017, 30. The three-dimensional point cloud instance segmentation network uses ISBNet in document 6: Ngo T D, Hua B S, Nguyen K. ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 13550-13559. In actual implementation, the loss function for training ISBNet is defined as follows:

[0123] L total = L cls + λ box L box + λ mask L mask + λ ms L MS

[0124] where L total represents the total loss, λ represents the weight of each loss, L cls represents the cross-entropy loss, L box represents the L1 and gIoU combined loss, L mask represents the dice and BCE combined loss, L MS represents the mask score loss. In actual implementation, λ box = 1, λ ms = 1, λ mask = 5.

[0125] Step 2.2, input the garden scene point cloud into the point cloud feature extraction network to obtain the point cloud feature F, which can be represented as:

[0126] F ∈ R N×D

[0127] where F represents the output feature, represents the real field, N represents the number of points, and D represents the output dimension.

[0128] Step 2.3, input the point cloud feature into the sampling-based instance-wise encoder and point-wise predictor in the instance segmentation network to extract the feature. The sampling-based instance-wise encoder mainly uses the instance-aware farthest point sampling layer and convolutional layer, which can be represented as:

[0129]

[0130] where P' represents the sampling point, K' represents the selected candidate point, represents the probability of the point as background, and τ is the hyperparameter threshold. The output of the sampling-based instance-wise encoder is the kernel prediction W ∈ R K×H' and the box prediction B ∈ R K×6' The output of the point-wise predictor is the mask feature F mask ∈ R N×H and the box feature F box ∈ R N×6 , where K is the number of instances, and H, H' are determined according to the convolution size. In the actual implementation process, the farthest point sampling layer is set as τ = 0.5, K = 384.

[0131] Step 2.4, connect the box feature F box and the box prediction B to obtain the geometric feature and the relative position feature , which are input into the box-aware dynamic convolution together with the mask feature. After the kernel prediction W is transformed to get the instance kernel w (k) , the processing can be represented as:

[0132]

[0133] where, represents the mask of the finally generated instance, Sigmoid represents the sigmoid function, and Conv represents the convolution operation. The box-aware dynamic convolution finally outputs the segmented mask, thereby obtaining the segmented instance, as shown in Figure 4 , where the masks of different gray levels represent different instances segmented out.

[0134] Step 3, generation of the instance relationship graph feature of the garden point cloud, which includes the following process:

[0135] Step 3.1, input the N instances segmented by the instance segmentation network into the linear layer in the instance relation graph generator to generate the node feature set in the garden relation graph where C m is the number of feature channels.

[0136] Step 3.2, input the three-dimensional point cloud and instances into linear layers respectively to obtain TxC h and Nx C h features, then obtain N x T and T x N feature matrices through feature fusion, and then obtain the weights of N x N edges through matrix multiplication, combined with the node feature set M g to obtain the garden instance relation graph As Figure 5 shown, an instance relation graph in a garden scene can be represented by the formula:

[0137]

[0138] where N represents the number of instances, T represents the number of connection relations, C h represents the number of feature channels, represents the garden instance relation graph, represents the instance node, ε represents the edge, M g represents the feature set of the node, and A represents the adjacency matrix.

[0139] Step 3.3, use the graph convolution operation in document 8: Huang S, Hui T, Liu S, et al. Referring image segmentation via cross-modal progressive comprehension [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 10488-10497. to process the garden instance relation graph, which can be represented by the formula:

[0140]

[0141] where, represents the feature after graph convolution, A represents the adjacency matrix, I represents the identity matrix, M g represents the feature set of the node, and W7 represents the learnable weight matrix.

[0142] Step 4, point cloud segmentation of large-scale garden point cloud, which includes the following process:

[0143] Step 4.1, train the point cloud segmentation network KPConvNet according to the literature 7: Thomas H, Qi C R, Deschaud J E, et al. Kpconv: Flexible and deformable convolution for point clouds [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6411-6420.

[0144] Step 4.2, input the large-scale garden scene point cloud P and the garden instance relationship graph feature F g Connect the input to the trained point cloud segmentation network KPConvNet to get the point-by-point class label, as shown in the following figure, different gray masks represent different class point cloud segmentation results. Figure 6

[0145] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the invention content of the large-scale garden scene point cloud segmentation method driven by the instance relationship provided by the present application and part or all of the steps in each embodiment when executed by the data processing unit. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0146] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be realized by means of a computer program and its corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a computer program, i.e. a software product, which can be stored in a storage medium, including a plurality of instructions for causing a device (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) containing a data processing unit to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0147] ​The application provides an example relationship-driven large-scale garden scene point cloud segmentation method, and there are many methods and approaches to specifically realize the technical scheme. The above description is only the preferred embodiment of the application, and it should be pointed out that, for ordinary skilled persons in the technical field, several improvements and refinements can be made without departing from the principle of the application, and these improvements and refinements should also be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be realized by using the prior art.

Claims

1. An example relation-driven large-scale garden scene point cloud segmentation method, characterized in that, Comprising the following steps: Step 1, instance annotation of the garden scene point cloud: for the garden scene point cloud data, first, the data density is reduced by using the method of downsampling, then the instance objects in the point cloud are clustered out by using the method of Euclidean clustering, and finally different instances are labeled according to the point cloud category; Step 2, instance segmentation of the garden scene point cloud: using the instance annotated garden scene point cloud data to train the instance segmentation network, and then using the instance segmentation network to extract instances from the garden scene point cloud; Step 3, generating instance relationship graph features of the garden scene point cloud: according to the obtained instances in the garden scene point cloud, combining the garden scene point cloud, generating the instance relationship graph in the garden scene point cloud, and obtaining the relationship graph features of the garden scene point cloud through graph convolution; Step 4, point cloud segmentation of the garden scene point cloud: connecting the garden scene point cloud and the relationship graph features to input into the trained point cloud segmentation network to obtain the point-by-point category information of the garden scene point cloud, and completing the instance relationship driven large-scale garden scene point cloud segmentation; Wherein, the generating instance relationship graph features of the garden scene point cloud in step 3 comprises the following steps: Step 3-1, input the instance segmented by the instance segmentation network into a linear layer in the instance relation graph generator to generate a node feature set in the garden relation graph wherein represents the number of feature channels​ Step 3-2, input the garden scene point cloud and the above instance into the linear layer in the relation graph generator respectively, obtain The weights of the edges, in combination with the set of node features Obtain the garden instance relation graph ; Step 3-3, processing the garden instance relation graph using a graph convolution operation to obtain garden instance relation graph features , and is specifically represented as: ； wherein, represents a feature passed through graph convolution, represents an identity matrix, represents a learnable weight matrix.

2. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 1, wherein, The instance annotation of the garden scene point cloud in step 1 comprises the following steps: Step 1-1, for the garden scene point cloud to be segmented wherein represents N three-dimensional coordinates belonging to the real number field, uniformly down-sampled to the down-sampled point cloud by using the farthest point sampling method The specific method is as follows: ； wherein, represent a point set, a down-sampled point cloud from a point set comprises, represent a sample point set, i.e. original points in the garden scene point cloud, represent the point farthest from the query point in the sample point set, represent the query point in the sample point set, represent a down-sampled region in the garden scene point cloud, represent the total number of points in the down-sampled region; Step 1-2, to the down-sampled point cloud The point cloud is clustered into different clusters using a preset Euclidean distance as the clustering threshold The point cloud is clustered into different clusters using a preset Euclidean distance as the clustering threshold The Euclidean distance of the points in the point cloud is calculated as follows: ； wherein, representing two points in the point cloud, representing coordinates of the two points, representing a total dimension of the point cloud, denotes the corresponding dimension; Step 1-3, first, the number of points clustered in step 1-2 is between and clusters, according to shape division, determine whether there is a connection relationship between the same shape clusters and clusters, and accordingly divide into different instance objects; wherein, is a preset minimum number of points for clustering, is a preset maximum number of points for clustering; Then in the cluster clustered in step 1-2, points greater than the minimum number of points in the cluster are divided into different instance objects according to the original point cloud annotation; points less than the minimum number of points in the cluster are spliced into instance objects if they can be spliced into instance objects through connection relationship, otherwise marked as non-instance objects, to obtain instance object annotation data.

3. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 2, wherein, The instance segmentation of the garden scene point cloud in step 2 comprises the following steps: Step 2-1, using the instance annotation data in step 1 to train the point cloud feature extraction network and the instance segmentation network, the instance segmentation network being a three-dimensional point cloud instance segmentation network of instance-aware sampling and box-aware dynamic convolution, using cross-entropy loss , L1 and gIoU combined loss, dice and BCE combined loss and mask score loss , and the total loss is obtained by combination , the total loss is used as the loss function to train the point cloud feature extraction network and the instance segmentation network; Step 2-2, input the garden scene point cloud into the trained point cloud feature extraction network to obtain point cloud features wherein, represents a real number field, represents the number of points, represents the output dimension; Step 2-3, the point cloud features are input into the trained instance segmentation network to obtain segmented instance objects.

4. The instance relationship-driven large-scale garden scene point cloud segmentation method according to claim 3, wherein, The obtaining segmented instance objects in step 2-3 comprises the following steps: Step 2-3-1, extracting features from the input point cloud The input trained instance segmentation network includes a sampled instance encoder and a point predictor, the instance encoder outputs core predictions and box predictions , and the point predictor outputs mask features and box features ; Step 2-3-2, output features of the instance encoder and output features of the point predictor input box-aware dynamic convolution, output instance segmentation results.

5. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 4, wherein, The point cloud segmentation of the garden scene point cloud in step 4 comprises the following steps: Step 4-1, training a point cloud segmentation network; Step 4-2, the garden scene point cloud and garden instance graph features connected and input to the trained point cloud segmentation network to obtain a point-by-point class label, completing instance relationship driven garden scene point cloud segmentation.

6. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 5, wherein, The method of Euclidean clustering in step 1-2 specifically comprises: First, the Euclidean distance between two points in space is calculated The distance is then compared to a set threshold, and points less than the threshold are clustered into a cluster, and points greater than the threshold are clustered into another cluster.

7. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 6, wherein, The total loss in step 2-1 is represented as follows: ； wherein, represents the total loss, , and represent the weights of the losses , and , represents the cross-entropy loss, represents the L1 and gIoU combined loss, represents the dice and BCE combined loss, represents the mask score loss.

8. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 7, wherein, The processing process of the box perception dynamic convolution in step 2-3-2 is as follows: ; wherein, a mask representing the last generated instance, i.e., instance segmentation result, represents a sigmoid function, represents a convolution operation, represents different features, represents an instance kernel.

9. The example relation-driven large-scale garden scene point cloud segmentation method according to claim 8, wherein, Garden instance relationship graph described in Step 3-2 is represented as: ； wherein, a representative landscape instance relationship graph, a representative instance node, a representative edge, a representative set of features of a node, a representative adjacency matrix.

Images