Point cloud segmentation method based on deep learning

Abstract

The invention discloses a point cloud segmentation method based on deep learning, and relates to the fields of three-dimensional vision, machine learning and the like. In order to solve the problems that point cloud of a stock ground cannot be accurately segmented and bulk cargo equipment cannot accurately sense stock pile information in the intelligent process of the bulk cargo equipment, firstly, data acquisition and preprocessing are carried out; secondly, point features are aggregated to form super points; then, designing a query strategy and a Transform decoder layer, and predicting an instance mask and a semantic category; and finally, designing a multi-task loss function and a matching strategy. Compared with the prior art, three segmentation tasks are completed through training of one instance segmentation data set, the positions and boundaries of different material piles in the stockyard can be accurately identified, the material categories of the different material piles can be classified, the warehouse management efficiency and the intelligent level of the stockyard can be remarkably improved, and the management efficiency of the stockyard is improved. Technical support is provided for efficient operation of bulk material equipment, and the method can be widely applied to the fields of bulk material equipment management, mineral collection, wharf carrying and the like.

Description

Technical Field

[0001] The present invention relates to fields such as three-dimensional vision and machine learning, and specifically to a point cloud segmentation method based on deep learning. Background Art

[0002] Large bulk material equipment plays an important role in industrial production, and its intelligent upgrade is an inevitable trend of future development. Intelligent bulk material equipment can achieve continuous operation, meet the needs of large-scale bulk material storage and reclaiming, and support a variety of stacking methods and can handle a variety of bulk materials. This upgrade can not only significantly reduce labor costs, but also reduce equipment downtime and improve production efficiency and safety.

[0003] In recent years, with the introduction of technologies such as the Internet of Things, big data, and artificial intelligence, the intelligence level of bulk material equipment has been continuously improved. Three-dimensional point cloud segmentation is one of the key technologies to realize the intelligence of large-scale bulk material equipment. By effectively segmenting point cloud data, it can provide an accurate basis for subsequent data processing and analysis, such as calculating the volume and weight of the pile, optimizing the pile-retrieving path, etc. However, point cloud segmentation faces many challenges in complex scenarios, especially when dealing with bulk material piles, it is necessary to accurately segment the pile from the messy point cloud.

[0004] At present, some significant progress has been made in point cloud segmentation algorithms. PointNet++ has improved local feature extraction, and Graph Convolutional Network (GCN) can process local and global features of point clouds. However, there are still problems such as high computational complexity and poor segmentation effect in complex scenes. It is unable to handle the three segmentation tasks of semantic segmentation, instance segmentation and panorama segmentation at the same time, and it is difficult to capture complex features, which limits its application in actual complex environments. At the same time, the existing algorithms are not suitable for use in environments such as material yards and mines. Due to the large number of dynamic objects and the high similarity of point clouds of different types of materials, point cloud segmentation is difficult, so that large bulk material equipment cannot identify the material pile to realize automatic material collection, and it is also impossible to calculate the volume and weight of the material pile, and it is impossible to realize the intelligent management of material yard data, and it cannot meet the further development needs of large bulk material equipment intelligence.

[0005] In view of the above problems and practical needs, the present invention designs a point cloud segmentation method based on deep learning to solve the problem that the stockpile point cloud in the material yard of large bulk material equipment cannot be accurately segmented. By training the data set with stockpile instance labels, the three segmentation tasks of semantic, instance and panoramic can be realized, avoiding the need to spend three times the time and training amount to complete the three segmentation tasks, and finally accurately segmenting each group of stockpile point clouds, and predicting the material category of each group of stockpile point clouds by deriving the instance mask. Summary of the invention

[0006] The present invention relates to a point cloud segmentation method based on deep learning. The method first obtains the point cloud data of a stockpile in a material yard of large-scale bulk material equipment, and then performs operations such as noise reduction, abnormal point removal and labeling. Then, the weights between points are designed, a local neighborhood graph is constructed through geometric and color similarities, and curvature features are introduced. 3D sparse convolution is used to extract point features, and super points are generated through clustering. Dynamic weighted average pooling is used to aggregate super point features. Then, an initial query vector is generated through task-conditional query, and a Transformer decoder is used to realize multi-scale feature interaction, and a convolution kernel is dynamically generated to support semantic segmentation, instance segmentation and panoramic segmentation multi-task joint output. Finally, multi-task losses are jointly optimized, and curvature losses are introduced for geometric regularization. The matching between predictions and real instances is simplified through a disentangled matching strategy, and the end-to-end training efficiency is optimized.

[0007] The specific implementation steps are as follows:

[0008] Step 1: Obtain and preprocess the environmental point cloud dataset of the large bulk material equipment operation area.

[0009] Step 1.1: Get the material field point cloud dataset that needs to be segmented. The original point cloud parameters need to include coordinate parameters and convert them into x, y, z with RGB color information.

[0010] Step 1.2: De-noise and remove outliers from the original point cloud, and annotate the stockpile point cloud, which includes instance segmentation datasets of different categories and their different individual labels.

[0011] Step 1.3: Data enhancement: increase data diversity and improve model generalization ability through operations such as random rotation and random flipping.

[0012] Step 2: Due to the large number of point clouds, direct training of the original point cloud will consume a lot of resources, and the curvature of the pile point cloud is an important feature. Therefore, this method adopts the method of adding curvature features to the original point cloud to enhance the effect of point cloud segmentation, and then forms the point cloud into super points, which greatly reduces the number of point clouds entering the pooling layer and the resources consumed by training.

[0013] Step 2.1: Construct a neighborhood. For each point , use an edge weight to divide points with similar geometric features and colors into the same neighborhood. The specific design steps are as follows:

[0014] ,

[0015] in, represents the minimum weight threshold; Indicate point and Point The specific expression of the weight value between is:

[0016] ,

[0017] in, is the equilibrium parameter, Yes and Point The geometric similarity between Yes and Point The color similarity is converted into weights through the Gaussian kernel function:

[0018] ,

[0019] ,

[0020] in, is the square of the Euclidean distance, Yes The color vector, and is a normalization parameter that controls the range of distance attenuation.

[0021] Step 2.2: Calculate the curvature. The specific steps are as follows:

[0022] Step 1: From the neighborhood constructed above, calculate the covariance matrix C of the neighborhood points:

[0023] ,

[0024] in, yes The total number of points in the point's neighborhood, is the centroid of the neighborhood points, is the distance weight between the neighborhood point and the center point, and its expression is:

[0025] ,

[0026] in, is the scale parameter.

[0027] Step 2: Curvature calculation, eigenvalue decomposition of the covariance matrix C, and obtain the eigenvalue , curvature By minimum eigenvalue calculate:

[0028] ,

[0029] Step 3: Preprocessing of curvature features, normalizing the curvature value to [ − 1 , 1 ] range to avoid numerical instability, and assign higher weights to points with larger curvature to highlight their importance;

[0030] Step 4: Add a curvature value to each point. If N points are input, it is expressed as , point cloud features from N×6 becomes N×7 ,get ,in, , , is the coordinate, is the curvature, , , Red, green and blue respectively, ranging from 0 to 255.

[0031] Step 2.3: Superpoint generation and feature extraction, first use 3D sparse convolution to Extract point features to get , and then use connectivity-based clustering to aggregate superpoints, the purpose is to merge points with similar characteristics into the same superpoint and generate a superpoint set , where the number of superpoints , super point The features of the precomputed super points The dynamic weighted average pooling formula is:

[0032] ,

[0033] in yes The characteristic vector of is the number of point clouds in the super-point area, For point The weight of Calculate weights:

[0034] ,

[0035] in is the weight parameter used to control the influence of curvature, is the curvature of the superpoint, yes The characteristic vector of A multi-layer perceptron.

[0036] Step 3: Design a query decoder and take two sets of queries as input to the query decoder. The number of queries is and ,in, represents the number of instance queries, and Represents the number of semantic queries. The core function of the query decoder is to convert these queries into an equal number of kernels, which dynamically adjust the network behavior by generating cross-scale feature interactions through task-conditioned query generation, without the need to design independent branches for different tasks.

[0037] Step 3.1: Task-Conditioned Query Initialization,The input of the query decoder is a task-specific query vector, and its initialization process combines the task semantics and the geometric features of the point cloud:

[0038] Step 1: Each task (semantic segmentation, instance segmentation, panoptic segmentation) is assigned a learnable task token , as the condition information of the task, where is the number of task types;

[0039] Step 2: Generate the initial query vector through the interaction of task tokens and global features:

[0040] ,

[0041] in, is the global feature obtained by pooling the highest level features of the sparse convolutional encoder, where is the superpoint feature dimension, It is a splicing operation. It is a multi-layer perceptron.

[0042] Step 3.2: The query decoder interacts with the multi-scale feature maps through the Transformer decoder layer to dynamically fuse local and global information:

[0043] Step 1: Get the multi-resolution feature map of the encoder ,in , For the The number of points of the layer, is the dimension of the projection matrix;

[0044] Step 2: Use the cross attention mechanism to convert the query vector With each scale feature map Interact and extract task-related features:

[0045] ,

[0046] in, is a learnable projection matrix, is the dimension of the projection matrix;

[0047] Step 3: Feature fusion, fuse the results of multi-scale cross attention to generate the final query vector:

[0048] ,

[0049] in, is a learnable scale weight, calculated via the attention mechanism.

[0050] Step 3.3: Dynamically generate convolution kernels through query features and directly predict segmentation results. Generate the corresponding convolution kernel ( is the convolution kernel size, is the above projection matrix dimension), its expression is:

[0051] ,

[0052] The convolution kernel is then applied to the multi-scale feature map to generate a mask:

[0053] ,

[0054] in, Represents the convolution operation, output For task-specific segmentation results, the convolution operation enables the kernel to capture useful information in the super-point features, thereby generating two sets of outputs: Instances and The instances represent independent objects or entities in the data. They are segmented from the original point cloud by the decoder, and the semantic masks are used to identify the semantic categories of these instances and determine what type of objects they belong to.

[0055] Step 4: This design defines a cost function between the query and the actual truth object, designs a matching strategy that minimizes this cost function, and formulates a loss function applied to the matching pairs to implement a transformer-based end-to-end training method.

[0056] Step 4.1: Design the cost function , used to measure the The prediction and The similarity of the actual true value is as follows:

[0057] ,

[0058] in, To measure the The prediction belongs to The probability of a semantic category, is an adjustable weight parameter, overlapping point mask matching cost It is the sum of binary cross entropy (BCE) and Dice loss with Laplace smoothing:

[0059] ,

[0060] in, They are the predicted truth mask and the actual truth mask of an overlapping point respectively.

[0061] Step 4.2: Disentangled matching, establish a correspondence between an actual truth object, a superpoint, an instance query and a guessed instance derived from this instance query, skip the intermediate correspondence, directly match the predicted instance to a real instance, disentangle the guessed value and the actual value, but the number of predictions still exceeds the number of actual truth instances, so it is necessary to filter out the predictions that do not correspond to the actual truth objects. The disentangled matching technique simplifies the cost function optimization and sets the largest weight in the cost matrix to infinity:

[0062] ,

[0063] Step 4.3: Design the total loss function , which is:

[0064] ,

[0065] in, is a weight parameter used to balance the importance of each part of the loss. The loss function of semantic segmentation is the cross entropy loss:

[0066] ,

[0067] in is the total number of super points, C is the total number of categories, It is The point belongs to the true label of category c, It is The predicted probability that a point is classified into category c; the loss function for instance segmentation is the binary cross entropy loss The sum of the Dice loss and the binary cross entropy loss is used to predict whether each point belongs to an instance, and its expression is:

[0068] ,

[0069] in is the instance label of the i-th point, is the instance probability predicted by the model, is the number of superpoints; the Dice loss function is used to calculate the overlap of instance masks and improve segmentation accuracy:

[0070] ,

[0071] in is the instance label of the i-th point, is the instance probability predicted by the model, is the number of superpoints; then the curvature is used The loss is geometrically regularized to enhance the model's perception of the surface morphology. This loss ensures the curvature consistency of the model during the optimization process and reduces misclassification of boundary areas:

[0072] ,

[0073] in is the curvature predicted by the model, is the true curvature, is the number of super points.

[0074] The present invention specifically relates to a point cloud segmentation method based on deep learning, which is used to solve the problem that the stockpile point clouds in the material yard of large-scale bulk material equipment cannot be accurately and quickly segmented. First, a special neighborhood of the original point cloud is constructed, the curvature characteristics of each point in the neighborhood are calculated, and then the dynamic weighted average pooling is performed to form the characteristics of the super-point, which greatly reduces the computational complexity; then, a task-conditional query is used to generate a dynamically adjusted network behavior of cross-scale feature interaction, without the need to design independent branches for different tasks, and to design cost functions, loss functions and matching mechanisms for them, and finally each group of stockpile point clouds is accurately segmented, and the material category of each group of stockpile point clouds is predicted by deriving an instance mask. The technical solution of the present invention has the following beneficial technical effects:

[0075] 1. Through one training cycle and one instance segmentation dataset training, three types of segmentation tasks, semantic, instance and panoramic, are completed, which greatly reduces the model training time and computing cost, and reduces costs;

[0076] 2. The curvature feature highlights the geometric structure of edges and corners, and can accurately identify the shape, position and boundaries of different material piles in the material yard, and classify their material categories (such as coal, ore, etc.), providing accurate data support for inventory management and material scheduling;

[0077] 3. Accurately segment each group of material pile point clouds, and predict the material type of each group of point clouds through the final instance mask, calculate the weight of the material pile through volume and density, help realize the automated management of material yard inventory, reduce manual measurement errors, and improve the accuracy and timeliness of inventory data;

[0078] 4. By accurately dividing the material pile, the scheduling and transportation plan of materials can be optimized, the equipment idle rate can be reduced, the transportation efficiency can be improved, and the operating cost can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is the overall flow chart of the present invention;

[0080] Figure 2 It is a network structure diagram of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0082] Attached Figure 1 It is an overall flow chart of the present invention. This embodiment relates to a point cloud segmentation method based on deep learning. Taking the point cloud of a large bulk material equipment yard as an example, it specifically includes the following processes: data preprocessing, super point generation and feature extraction, design of query decoder, and design of loss function and matching mechanism.

[0083] Attached Figure 2 It is a network structure diagram of the present invention, which has the structure of input data, data preprocessing, sparse 3D convolution layer, average pooling layer output super-point features, the value, key and query selection output of the super-point feature are input to the Transformer decoder layer, and then the instance kernel, instance score and semantic kernel of the point cloud are output through the disentanglement matching mechanism to compare with the super-point features, and finally the three segmentation results of semantic, panoramic and instance are output.

[0084] The specific implementation instructions are as follows:

[0085] Step 1: Obtain and preprocess the environmental point cloud dataset of the large bulk material equipment operation area.

[0086] Step 1.1: Use 3D LiDAR and depth camera to collect point cloud data of bulk material in the silo, ensure that the data contains the 3D coordinates (x, y, z) and RGB color information (0-255) of each point, deploy real-time scanning equipment and set regular scanning tasks.

[0087] Step 1.2: Use statistical outlier removal to remove outliers, then use Gaussian filtering to smooth the noise, and then use the CloudCompare tool to assign a unique instance label and semantic category (such as coal, ore, sand and gravel, etc.) to each stockpile instance, and finally divide the data set (70% training set, 15% validation set, and 15% test set).

[0088] Step 1.3: Data enhancement: perform random rotation (±10°), random translation (±0.5m), random scaling (0.9~1.1 times) and random color perturbation (±20% brightness) on the point cloud to increase data diversity.

[0089] Step 2: Due to the large number of point clouds, direct training of the original point cloud will consume a lot of resources, and the curvature of the pile point cloud is an important feature. Therefore, this method adopts the method of adding curvature features to the original point cloud to enhance the effect of point cloud segmentation, and then forms the point cloud into super points, which greatly reduces the number of point clouds entering the pooling layer and the resources consumed by training.

[0090] Step 2.1: Construct a neighborhood. For each point , use an edge weight to divide points with similar geometric features and colors into the same neighborhood. The specific design steps are as follows:

[0091] ,

[0092] in, represents the minimum weight threshold; Indicate point and Point The specific expression of the weight value between is:

[0093] ,

[0094] in, is the equilibrium parameter, Yes and Point The geometric similarity between Yes and Point The color similarity is converted into weights through the Gaussian kernel function:

[0095] ,

[0096] ,

[0097] in, is the square of the Euclidean distance, Yes The color vector, and is a normalization parameter that controls the range of distance attenuation.

[0098] Step 2.2: Calculate the curvature. The specific steps are as follows:

[0099] Step 1: From the neighborhood constructed above, calculate the covariance matrix C of the neighborhood points:

[0100] ,

[0101] in, yes The total number of points in the point's neighborhood, is the centroid of the neighborhood points, is the distance weight between the neighborhood point and the center point, and its expression is:

[0102] ,

[0103] in, is the scale parameter.

[0104] Step 2: Curvature calculation, eigenvalue decomposition of the covariance matrix C, and obtain the eigenvalue , curvature By minimum eigenvalue calculate:

[0105] ,

[0106] Step 3: Preprocessing of curvature features, normalizing the curvature value to [ − 1 , 1 ] range to avoid numerical instability, and assign higher weights to points with larger curvature to highlight their importance;

[0107] Step 4: Add a curvature value to each point. If N points are input, it is expressed as , point cloud features from N×6 becomes N×7 ,get ,in, , , is the coordinate, is the curvature, , , They are red, green, and blue, ranging from 0 to 255.

[0108] Step 2.3: Superpoint generation and feature extraction, first use 3D sparse convolution to Extract point features to get , and then use Euclidean clustering to aggregate superpoints, the purpose is to merge points with similar characteristics into the same superpoint and generate a superpoint set , where the number of superpoints , super point The features of the precomputed super points The dynamic weighted average pooling formula is:

[0109] ,

[0110] in yes The characteristic vector of is the number of point clouds in the super-point area, For point The weight of Calculate weights:

[0111] ,

[0112] in is the weight parameter used to control the influence of curvature, is the curvature of the superpoint, yes The characteristic vector of A multi-layer perceptron.

[0113] Step 3: Design a query decoder and take two sets of queries as input to the query decoder. The number of queries is and ,in, represents the number of instance queries, and Represents the number of semantic queries. The core function of the query decoder is to convert these queries into an equal number of kernels, which dynamically adjust the network behavior by generating cross-scale feature interactions through task-conditioned query generation, without the need to design independent branches for different tasks.

[0114] Step 3.1: Task-Conditioned Query Initialization,The input of the query decoder is a task-specific query vector, and its initialization process combines the task semantics and the geometric features of the point cloud:

[0115] Step 1: Each task (semantic segmentation, instance segmentation, panoptic segmentation) is assigned a learnable task token , as the condition information of the task, where is the number of task types;

[0116] Step 2: Generate the initial query vector through the interaction of task tokens and global features:

[0117] ,

[0118] in, is the global feature obtained by pooling the highest level features of the sparse convolutional encoder, where is the superpoint feature dimension, It is a splicing operation. It is a multi-layer perceptron.

[0119] Step 3.2: The query decoder interacts with the multi-scale feature maps through the Transformer decoder layer to dynamically fuse local and global information:

[0120] Step 1: Get the multi-resolution feature map of the encoder ,in , For the The number of points of the layer, is the dimension of the projection matrix;

[0121] Step 2: Use the cross attention mechanism to convert the query vector With each scale feature map Interact and extract task-related features:

[0122] ,

[0123] in, is a learnable projection matrix, is the dimension of the projection matrix;

[0124] Step 3: Feature fusion, fuse the results of multi-scale cross attention to generate the final query vector:

[0125] ,

[0126] in, is a learnable scale weight, calculated via the attention mechanism.

[0127] Step 3.3: Dynamically generate convolution kernels through query features and directly predict segmentation results. Generate the corresponding convolution kernel ( is the convolution kernel size, is the above projection matrix dimension), its expression is:

[0128] ,

[0129] The convolution kernel is then applied to the multi-scale feature map to generate a mask:

[0130] ,

[0131] in, Represents the convolution operation, output For task-specific segmentation results, the convolution operation enables the kernel to capture useful information in the super-point features, thereby generating two sets of outputs: Instances and The instances represent independent objects or entities in the data. They are segmented from the original point cloud by the decoder, and the semantic masks are used to identify the semantic categories of these instances and determine what type of objects they belong to.

[0132] Step 4: This design defines a cost function between the query and the actual truth object, designs a matching strategy that minimizes this cost function, and formulates a loss function applied to the matching pairs to implement a transformer-based end-to-end training method.

[0133] Step 4.1: Design the cost function , used to measure the The prediction and The similarity of the actual true value is as follows:

[0134] ,

[0135] in, To measure the The prediction belongs to The probability of a semantic category, is an adjustable weight parameter, overlapping point mask matching cost It is the sum of binary cross entropy (BCE) and Dice loss with Laplace smoothing:

[0136] ,

[0137] in, They are the predicted truth mask and the actual truth mask of an overlapping point respectively.

[0138] Step 4.2: Disentangled matching, establish a correspondence between an actual truth object, a superpoint, an instance query and a guessed instance derived from this instance query, skip the intermediate correspondence, directly match the predicted instance to a real instance, disentangle the guessed value and the actual value, but the number of predictions still exceeds the number of actual truth instances, so it is necessary to filter out the predictions that do not correspond to the actual truth objects. The disentangled matching technique simplifies the cost function optimization and sets the largest weight in the cost matrix to infinity:

[0139] ,

[0140] Step 4.3: Design the total loss function , which is:

[0141] ,

[0142] in, is a weight parameter used to balance the importance of each part of the loss. The loss function of semantic segmentation is the cross entropy loss:

[0143] ,

[0144] in is the total number of super points, C is the total number of categories, It is The point belongs to the true label of category c, It is The predicted probability that a point is classified into category c; the loss function of instance segmentation is the sum of binary cross entropy loss and Dice loss. Binary cross entropy loss is used to predict whether each point belongs to a certain instance, and its expression is:

[0145] ,

[0146] in is the instance label of the i-th point, is the instance probability predicted by the model, is the number of superpoints; the Dice loss function is used to calculate the overlap of instance masks and improve segmentation accuracy:

[0147] ,

[0148] in is the instance label of the i-th point, is the instance probability predicted by the model, is the number of superpoints; then the curvature is used The loss is geometrically regularized to enhance the model's perception of the surface morphology. This loss ensures the curvature consistency of the model during the optimization process and reduces misclassification of boundary areas:

[0149] ,

[0150] in is the curvature predicted by the model, is the true curvature, is the number of super points.

[0151] Through the above steps, a neural network model can be built that can perform semantic segmentation, instance segmentation and panoramic segmentation on three-dimensional point clouds. The divided data is then put into the model for training, and each pile of material point cloud is finally segmented accurately. The material type of each pile of material point cloud is predicted by the instance mask obtained at the end.

Claims

1. A point cloud segmentation method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain the environmental point cloud dataset of the large bulk material equipment operation area and preprocess it; Step 2: Construct a neighborhood and calculate the curvature of the points in the neighborhood and the curvature-weighted super-point features; Step 3: Design the decoder, initialize the query vector, and select the query selection strategy; Step 4: Design cost function, loss function and disentanglement matching strategy.

2. The point cloud segmentation method based on deep learning according to claim 1, characterized in that: The steps of constructing the neighborhood described in step 2 and calculating the curvature of the point in the neighborhood and the curvature-weighted super-point features are as follows: Step 2.1: Build the neighborhood , for each point : ， in, represents the minimum weight threshold; Indicate point and Point The specific expression of the weight value between is: ， in, is the equilibrium parameter, Yes and Point The geometric similarity between Yes and Point The color similarity is converted into weights through the Gaussian kernel function: ， ， in, is the square of the Euclidean distance, Yes The color vector, and is the normalization parameter; Step 2.2: Based on the neighborhood constructed above, calculate the curvature of each point through the covariance matrix , and added to the point features, the input point cloud is represented as , is the number of point clouds, and the number of features of point clouds is 6, which are , are coordinates, are red, green and blue respectively, and their range is , adding curvature After that, we get ; Step 2.3: Superpoint generation and feature extraction, first use 3D sparse convolution to Extract point features to get , and then use the clustering algorithm based on connectivity to aggregate superpoints to form a superpoint domain, merge points with similar characteristics into the same superpoint, and generate a superpoint set , where the number of superpoints , super point The characteristic is obtained by dynamically weighted average pooling of the point cloud in each super-point domain. The dynamic weighted average pooling formula is: ， in yes The characteristic vector of is the number of point clouds in the super-point area, For point The weight of Calculate weights: ， in is the weight parameter, is the curvature of the point cloud, yes The characteristic vector of A multi-layer perceptron.

3. The point cloud segmentation method based on deep learning according to claim 1, characterized in that: The steps of designing the decoder, initializing the query vector, and selecting the query selection strategy described in step 3 are as follows: Step 3.1: Task conditional query initialization: Step 1: Assign a task token to each task , as the condition information of the task, where is the number of task types; Step 2: Generate the initial query vector through the interaction of task tokens and global features: ， in, is the global feature obtained by pooling the highest-level features of the sparse convolutional encoder, where is the superpoint feature dimension, It is a splicing operation. It is a multi-layer perceptron; Step 3.2: Design the query decoder: Step 1: Get the multi-resolution feature map of the encoder ,in , For the The number of points of the layer, is the dimension of the superpoint feature; Step 2: Use the cross attention mechanism to transform the query vector With each scale feature map To interact: ， in, is the projection matrix, is the dimension of the projection matrix; Step 3: Final query vector expression: ， in, is the scale weight; Step 3.3: Each query Generate the corresponding convolution kernel ,in is the convolution kernel size, is the dimension of the above projection matrix, and its expression is: ， Then convolve the convolution kernel with the multi-scale feature map to obtain the mask : ， in, represents the convolution operation, is the convolution kernel.

4. The point cloud segmentation method based on deep learning according to claim 1, characterized in that: The design cost function, loss function and disentanglement matching strategy described in step 4 are as follows: Step 4.1: Design the cost function ; Step 4.2: Disentanglement Matching , when Point belongs to When there are objects, the cost is , otherwise it is infinite: ， Step 4.3: Design the total loss function : ， in, is the weight parameter, the loss function of semantic segmentation is the cross entropy loss: ， in is the total number of super points, C is the total number of categories, It is The point belongs to the true label of category c, It is The predicted probability that a point is classified into category c; the loss function for instance segmentation is the binary cross entropy loss The sum of the Dice loss and the binary cross entropy loss is expressed as: ， in is the instance label of the i-th point, is the instance probability predicted by the model, is the number of super points; Dice loss function: ， in is the instance label of the i-th point, is the instance probability predicted by the model, is the number of superpoints; then the curvature The loss is geometrically regularized: ， in is the curvature predicted by the model, is the true curvature, is the number of super points.

Images