Point cloud instance segmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a point cloud instance segmentation method and device, electronic equipment and a storage medium, and relates to the technical field of computer data processing, and the method comprises the steps: carrying out the preprocessing of original point cloud data, and obtaining target point cloud data; obtaining an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least comprises an initial global feature extraction network and an initial local feature extraction network; inputting the target point cloud data into an initial point cloud segmentation model, extracting a first global feature corresponding to the target point cloud data through an initial global feature extraction network, and extracting a first local feature corresponding to the target point cloud data through an initial local feature extraction network; fusing the first global feature and the first local feature to obtain a target fusion feature; and training the initial point cloud segmentation model according to the target fusion features to generate a target point cloud segmentation model, thereby improving data processing efficiency, and improving accuracy and robustness of point cloud instance segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a point cloud instance segmentation method, a point cloud instance segmentation device, an electronic device and a computer-readable storage medium. Background Art

[0002] PointNet is a deep learning model that directly processes point cloud data. It extracts features from each point through a multi-layer perceptron (MLP) and uses a maximum pooling layer to obtain global features, thereby achieving classification and segmentation of point clouds. However, PointNet still has certain limitations for point cloud instance segmentation tasks, especially when dealing with complex scenes and similar objects, its performance will be affected to a certain extent.

[0003] Point cloud data can provide rich three-dimensional spatial information, making it an important data source for many application fields. Point cloud instance segmentation is a key pre-processing step, which can identify and distinguish different objects in the scene. However, in the process of point cloud instance segmentation, it relies on a large amount of manually labeled data, which is not only inefficient but also costly. In addition, the subjective errors and deviations that may be introduced in the acquisition and processing of labeled data also limit the performance of the algorithm. Summary of the invention

[0004] The embodiments of the present invention provide a point cloud instance segmentation method, device, electronic device and computer-readable storage medium to solve or partially solve the problems of low efficiency, high cost and low segmentation accuracy when segmenting point clouds.

[0005] An embodiment of the present invention discloses a point cloud instance segmentation method, comprising:

[0006] Preprocess the original point cloud data to obtain target point cloud data;

[0007] Acquire an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network;

[0008] Inputting the target point cloud data into the initial point cloud segmentation model, extracting a first global feature corresponding to the target point cloud data through the initial global feature extraction network, and extracting a first local feature corresponding to the target point cloud data through the initial local feature extraction network in parallel;

[0009] Fusing the first global feature with the first local feature to obtain a target fused feature;

[0010] The initial point cloud segmentation model is trained according to the target fusion features to generate a target point cloud segmentation model, and the target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances.

[0011] In some feasible implementations, fusing the global feature with the local feature to obtain a target fused feature includes:

[0012] Concatenate the first global feature and the first local feature to obtain an initial fusion feature;

[0013] Inputting the initial fusion features into a preset learning model for self-supervised training to obtain a contrast loss function corresponding to the initial fusion model;

[0014] Based on the contrast loss function, the initial global feature extraction network and the initial local feature extraction network are parameter-tuned to obtain a target global feature extraction network and a target local feature extraction network;

[0015] The second global feature corresponding to the target point cloud data is extracted through the target global feature extraction network, and the second local feature corresponding to the target point cloud data is extracted through the target local feature extraction network, and the second global feature is fused with the second local feature to obtain a target fusion feature.

[0016] In some possible implementations, for each point p in the point cloud instance i , the contrast loss function is:

[0017]

[0018] Among them, f fused (p i ), f fused (p j ) are the initial fusion features, sim(.,.) is the similarity measurement function, N + is the positive sample set, N - is the set of negative samples, L contrastive (p i ) is the contrast loss function.

[0019] In some feasible implementations, training the initial point cloud segmentation model according to the target fusion feature to generate a target point cloud segmentation model includes:

[0020] Determine enhancement parameters for the target fusion feature,

[0021] The target fusion feature is enhanced by using the enhancement parameter to obtain an enhanced fusion feature;

[0022] The enhanced fusion features are used to train the initial point cloud segmentation model to generate a target point cloud segmentation model.

[0023] In some feasible implementations, the enhancement parameters include at least the local density and geometric structure dynamics corresponding to the target point cloud data, and the use of the enhancement parameters to enhance the target fusion feature to obtain the enhanced fusion feature includes:

[0024] Dynamically determining the local density corresponding to the target point cloud data using the geometric structure;

[0025] The target fusion feature is enhanced by using the local density to obtain an enhanced fusion feature.

[0026] In some feasible implementations, the local density corresponding to the target point cloud data is dynamically determined using the geometric structure and is calculated using the following formula:

[0027]

[0028] Among them, the |N(p i )| is point p i The number of points in the neighborhood, the V(p i ) is the volume of the neighborhood, and the ρ(p i ) is the local density.

[0029] In some feasible implementations, the target fusion feature is enhanced by using the local density to obtain an enhanced fusion feature, which is calculated by the following enhancement calculation formula:

[0030] f′ fused (p i )=Enhance(f(p i ),ρ(p i ))

[0031] Among them, the f′ fused (p i ) is an enhanced fusion feature, and the f(p i ) is the target fusion feature, and the ρ(p i ) is the local density, and the Enhance function is based on the local density ρ(p i ) to enhance the target fusion vector.

[0032] In some feasible implementations, the adopting the enhanced fusion feature to train the initial point cloud segmentation model to generate a target point cloud segmentation model includes:

[0033] Inputting the enhanced fusion feature into the initial point cloud segmentation model for prediction, and obtaining a prediction result corresponding to the enhanced fusion feature;

[0034] Calculate the loss function between the prediction result and the preset true label;

[0035] The initial point cloud segmentation model is tuned using the loss function until the loss function between the predicted result and the true label meets the preset conditions, the model training is stopped, and the target point cloud segmentation model is obtained.

[0036] In some feasible implementations, the preprocessing of the original point cloud data to obtain the target point cloud data includes:

[0037] Acquire original point cloud data, where the original point cloud data includes a plurality of points;

[0038] Obtaining the neighborhood corresponding to each of the points;

[0039] Calculate the number of other points in the neighborhood corresponding to each point in the original point cloud data based on local density filtering, and take the target point whose number of other points in the neighborhood is less than a preset threshold as a noise point, and remove the noise point from the original point cloud data;

[0040] For the original point cloud data with noise points removed, the original point cloud data is sampled by uniform downsampling or voxel downsampling, and the sampled original point cloud data is spatially normalized to obtain target point cloud data.

[0041] The embodiment of the present invention further discloses a point cloud instance segmentation device, comprising:

[0042] Point cloud data processing module, used to pre-process the original point cloud data to obtain target point cloud data;

[0043] A model acquisition module, used to acquire an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network;

[0044] A feature extraction module, used for inputting the target point cloud data into the initial point cloud segmentation model, and extracting a first global feature corresponding to the target point cloud data through the initial global feature extraction network, and extracting a first local feature corresponding to the target point cloud data through the initial local feature extraction network;

[0045] A feature fusion module, used for fusing the first global feature with the first local feature to obtain a target fusion feature;

[0046] The model training module is used to train the initial point cloud segmentation model according to the target fusion feature to generate a target point cloud segmentation model, and the target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances.

[0047] In some feasible implementations, the feature fusion module is specifically used to:

[0048] Concatenate the first global feature and the first local feature to obtain an initial fusion feature;

[0049] Inputting the initial fusion features into a preset learning model for self-supervised training to obtain a contrast loss function corresponding to the initial fusion model;

[0050] Based on the contrast loss function, the initial global feature extraction network and the initial local feature extraction network are parameter-tuned to obtain a target global feature extraction network and a target local feature extraction network;

[0051] The second global feature corresponding to the target point cloud data is extracted through the target global feature extraction network, and the second local feature corresponding to the target point cloud data is extracted through the target local feature extraction network, and the second global feature is fused with the second local feature to obtain a target fusion feature.

[0052] In some possible implementations, for each point p in the point cloud instance i , the contrast loss function is:

[0053]

[0054] Among them, f fused (p i ), f fused (p j ) are the initial fusion features, sim(.,.) is the similarity measurement function, N + is the positive sample set, N - is the set of negative samples, L contrastive (p i ) is the contrast loss function.

[0055] In some feasible implementations, the model training module is specifically used to:

[0056] Determine enhancement parameters for the target fusion feature,

[0057] The target fusion feature is enhanced by using the enhancement parameter to obtain an enhanced fusion feature;

[0058] The enhanced fusion features are used to train the initial point cloud segmentation model to generate a target point cloud segmentation model.

[0059] In some feasible implementations, the enhancement parameters include at least the local density and geometric structure dynamics corresponding to the target point cloud data, and the model training module is specifically used for:

[0060] Dynamically determining the local density corresponding to the target point cloud data using the geometric structure;

[0061] The target fusion feature is enhanced by using the local density to obtain an enhanced fusion feature.

[0062] In some feasible implementations, the model training module is specifically used to perform calculations using the following formula:

[0063]

[0064] Among them, the |N(p i )| is point p i The number of points in the neighborhood, the V(p i ) is the volume of the neighborhood, and the ρ(p i ) is the local density.

[0065] In some feasible implementations, the model training module is specifically used to perform calculations using the following formula:

[0066] f′ fused (p i )=Enhance(f(p i ),ρ(p i ))

[0067] Among them, the f′ fused (p i ) is an enhanced fusion feature, and the f(p i ) is the target fusion feature, and the ρ(p i ) is the local density, and the Enhance function is based on the local density ρ(p i ) to enhance the target fusion vector.

[0068] In some feasible implementations, the model training module is specifically used to:

[0069] Inputting the enhanced fusion feature into the initial point cloud segmentation model for prediction, and obtaining a prediction result corresponding to the enhanced fusion feature;

[0070] Calculate the loss function between the prediction result and the preset true label;

[0071] The initial point cloud segmentation model is tuned using the loss function until the loss function between the predicted result and the true label meets the preset conditions, the model training is stopped, and the target point cloud segmentation model is obtained.

[0072] In some feasible implementations, the point cloud data processing module is specifically used to:

[0073] Acquire original point cloud data, where the original point cloud data includes a plurality of points;

[0074] Obtaining the neighborhood corresponding to each of the points;

[0075] Calculate the number of other points in the neighborhood corresponding to each point in the original point cloud data based on local density filtering, and take the target point whose number of other points in the neighborhood is less than a preset threshold as a noise point, and remove the noise point from the original point cloud data;

[0076] For the original point cloud data with noise points removed, the original point cloud data is sampled by uniform downsampling or voxel downsampling, and the sampled original point cloud data is spatially normalized to obtain target point cloud data.

[0077] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0078] The memory is used to store computer programs;

[0079] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0080] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0081] The embodiments of the present invention include the following advantages:

[0082] In an embodiment of the present invention, when performing instance segmentation on a point cloud, a corresponding point cloud segmentation model may be trained first, and then instance segmentation may be performed on the point cloud data based on the trained point cloud segmentation model. During the training process, the original point cloud data may be preprocessed first to obtain target point cloud data, and an initial point cloud segmentation model may be obtained. The initial point cloud segmentation model includes at least an initial global feature extraction network and an initial local feature extraction network. The target point cloud data is then input into the initial point cloud segmentation model, and a first global feature corresponding to the target point cloud data is extracted through the initial global feature extraction network in parallel, and a first local feature corresponding to the target point cloud data is extracted through the initial local feature extraction network. Then, the first local feature corresponding to the target point cloud data is extracted through the first local feature extraction network. A global feature is fused with the first local feature to obtain a target fusion feature, and finally the initial point cloud segmentation model is trained according to the target fusion feature to generate a target point cloud segmentation model. The target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances. Therefore, in the process of model training, the local feature extraction network and the global feature extraction network are used for parallel processing, and the outputs of the two are fused to obtain corresponding fusion features, which realizes the feature distinguishability of different instances in the learning point cloud data, improves the processing ability of the point cloud segmentation network for complex scenes, reduces the dependence on manually labeled data, improves data processing efficiency, and improves the accuracy and robustness of point cloud instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a flowchart of the steps of a point cloud instance segmentation method provided in an embodiment of the present invention;

[0084] Figure 2 is a schematic diagram of the model training process provided in an embodiment of the present invention;

[0085] Figure 3 is a schematic diagram of a self-supervised contrastive learning framework provided in an embodiment of the present invention;

[0086] Figure 4 is a schematic diagram of an adaptive feature enhancement strategy provided in an embodiment of the present invention;

[0087] Figure 5 It is a structural block diagram of a point cloud instance segmentation device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0089] As an example, in the process of point cloud instance segmentation, relying on a large amount of manually annotated data is not only inefficient but also costly. In addition, the subjective errors and biases that may be introduced in the acquisition and processing of annotated data also limit the performance of the algorithm.

[0090] In this regard, in the present invention, when performing instance segmentation on the point cloud, the corresponding point cloud segmentation model can be trained first, and then the point cloud data can be instance segmented based on the trained point cloud segmentation model. During the training process, the original point cloud data can be preprocessed first to obtain the target point cloud data, and the initial point cloud segmentation model can be obtained. The initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network. Then, the target point cloud data is input into the initial point cloud segmentation model, and the first global feature corresponding to the target point cloud data is extracted through the initial global feature extraction network, and the first local feature corresponding to the target point cloud data is extracted through the initial local feature extraction network. Then, the first local feature corresponding to the target point cloud data is extracted. A global feature is fused with the first local feature to obtain a target fusion feature, and finally the initial point cloud segmentation model is trained according to the target fusion feature to generate a target point cloud segmentation model. The target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances. Therefore, in the process of model training, the local feature extraction network and the global feature extraction network are used for parallel processing, and the outputs of the two are fused to obtain corresponding fusion features, which realizes the feature distinguishability of different instances in the learning point cloud data, improves the processing ability of the point cloud segmentation network for complex scenes, reduces the dependence on manually labeled data, improves data processing efficiency, and improves the accuracy and robustness of point cloud instance segmentation.

[0091] Reference Figure 1 , shows a flowchart of a point cloud instance segmentation method provided in an embodiment of the present invention, which may specifically include the following steps:

[0092] Step 101, preprocessing the original point cloud data to obtain target point cloud data;

[0093] In the embodiment of the present invention, point cloud data can provide rich three-dimensional spatial information. By performing instance segmentation on point cloud data, each point in the point cloud can be assigned to an instance of a specific object. For example, point cloud instance segmentation can be used to identify and segment vehicles, pedestrians, obstacles, etc. on the road, and can also be used to identify and segment objects in the environment, help robots navigate, detect and segment defects and anomalies in industrial products, and can also be used to identify and segment packages in the logistics process.

[0094] Among them, the corresponding point cloud data can be collected by laser radar, depth camera and other equipment. Each point in the point cloud can contain corresponding position information (such as three-dimensional coordinates (x, y, z), etc.) and possible other attributes (such as color, normal, etc.), which is not limited by the present invention.

[0095] In a specific implementation, after the original point cloud data is obtained, the original point cloud data can be preprocessed to remove irrelevant information and improve the quality of subsequent processing. Specifically, the original point cloud data includes several points, then the neighborhood corresponding to each point can be obtained first, and then the number of other points in the neighborhood corresponding to each point in the original point cloud data is calculated based on local density filtering, and the target point whose number of other points in the neighborhood is less than a preset threshold is taken as a noise point, and the noise point is removed from the original point cloud data, and then the original point cloud data with the noise point removed is sampled by uniform downsampling or voxel downsampling, and the sampled original point cloud data is spatially normalized to obtain the target point cloud data, so that the computational complexity can be effectively reduced through preprocessing, so that the distribution of point cloud data is more uniform, which is helpful for network training. Among them, the neighborhood corresponding to each point can be determined by calculating the distance between each point and its surrounding points. For example, the neighborhood corresponding to each point in the point cloud data can be determined by using K-Nearest Neighbors (KNN) and radius search, and then the number of other points in the neighborhood can be counted.

[0096] In one example, the neighborhood corresponding to each point can be determined by calculating the distance between each point and its surrounding points, and then the number of other points contained in the neighborhood corresponding to each point can be counted. Then, the number of other points in the neighborhood of each point is calculated based on local density filtering, and the target points whose number of other points in the neighborhood is less than a preset threshold are removed as noise points. For example, the threshold is set to 10, that is, if the number of other points in the neighborhood of a point is less than 10, the point is considered to be a noise point and the noise point is removed. For the original point cloud data with noise points removed, the original point cloud data is sampled by uniform downsampling or voxel downsampling to reduce the amount of data and computational complexity. For example, the voxel downsampling method is used to downsample the point cloud data to a voxel size of 0.05. Finally, the sampled point cloud data is spatially normalized to ensure that the data is at the same scale, which is helpful for network training. Therefore, through preprocessing, the computational complexity can be effectively reduced, and the distribution of point cloud data is made more uniform, which is helpful for network training.

[0097] It should be noted that spatial normalization can be a process of converting data to the same scale to ensure that different features or dimensions have the same importance. For example, the minimum-maximum normalization method can be used to scale the data to a fixed interval, usually [0, 1]. The corresponding implementation formula can be: x'=(x-xmin) / xmax-xmin), and the present invention is not limited to this.

[0098] Step 102, obtaining an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network;

[0099] After the target point cloud data is determined, an initial point cloud segmentation model can be further obtained to train the initial point cloud segmentation model to obtain a point cloud segmentation model that can be used to perform instance segmentation on the point cloud data. The initial point cloud segmentation model can at least include an initial global feature extraction network and an initial local feature extraction network. The initial global feature extraction network can be used to extract global features from the point cloud data, and the initial local feature extraction network can be used to extract local features from the point cloud data.

[0100] Step 103, inputting the target point cloud data into the initial point cloud segmentation model, and extracting the first global feature corresponding to the target point cloud data through the initial global feature extraction network, and extracting the first local feature corresponding to the target point cloud data through the initial local feature extraction network in parallel;

[0101] In a specific implementation, the target point cloud data can be used as training data and input into the initial point cloud segmentation model for model training. The feature extraction of the target point cloud data can be performed in parallel through the initial global feature extraction network and the initial local feature network. Specifically, the first global feature corresponding to the target point cloud data can be extracted through the initial global feature extraction network, and the first local feature corresponding to the target point cloud data can be extracted through the initial local feature extraction network. Thus, by extracting global features and local features, the feature types are enriched, which is beneficial to improving the accuracy of subsequent model training.

[0102] Among them, the global feature network can be a PointNet network, and the PointNet network can include an input layer, an MLP layer (multi-layer perceptron) and a maximum pooling layer. The target point cloud data can be input into the PointNet network, and the global features of the target point cloud data are extracted through the input layer, MLP layer and maximum pooling layer in the PointNet network to obtain the first global feature corresponding to the target point cloud data; the local feature network can be a local feature editor, and the local feature editor can include a neighborhood point sampling layer, a local feature extraction layer and a feature encoding layer. The target point cloud data can be input into the local feature editor, and the local features of the target point cloud data are extracted through the neighborhood point sampling layer, the local feature extraction layer and the feature encoding layer to obtain the corresponding first local feature, thereby enriching the feature types by extracting global features and local features, which is beneficial to improving the accuracy of subsequent model training.

[0103] Step 104, fusing the first global feature with the first local feature to obtain a target fused feature;

[0104] After the first global feature and the first local feature corresponding to the target point cloud data are extracted, the first global feature and the first local feature may be fused to obtain a target fused feature, that is, a single feature vector.

[0105] In a specific implementation, the initial point cloud segmentation model may also include a feature fusion layer, through which the first global feature and the first local feature can be spliced ​​to obtain an initial fusion feature, and then the initial fusion feature is input into a preset learning model for self-supervised training to obtain a contrast loss function corresponding to the initial fusion model, and then the initial global feature extraction network and the initial local feature extraction network are parameter tuned based on the contrast loss function to obtain a target global feature extraction network and a target local feature extraction network, and then the second global feature corresponding to the target point cloud data is extracted through the target global feature extraction network, and the second local feature corresponding to the target point cloud data is extracted through the target local feature extraction network, and the second global feature is fused with the second local feature to obtain the target fusion feature.

[0106] Optionally, for each point p in the point cloud instance i , the contrast loss function is:

[0107]

[0108] Among them, f fused (p i ), f fused (p j ) are the initial fusion features, sim(.,.) is the similarity measurement function, N + is the positive sample set, N- is the set of negative samples, L contrastive (p i ) is the contrast loss function.

[0109] It should be noted that in the process of contrastive learning integration, the fused feature vector can be input into the contrastive learning framework in the initial point cloud segmentation model for self-supervised training, so that by comparing the fused feature vectors during the self-supervised training process, the model can learn to distinguish the features between the same instance and different instances, so that the point cloud segmentation model can improve the accuracy of instance segmentation and improve the model's ability to handle complex instances.

[0110] Step 105 , training the initial point cloud segmentation model according to the target fusion features to generate a target point cloud segmentation model, wherein the target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances.

[0111] After obtaining the corresponding target fusion features, the initial point cloud segmentation model can be trained based on the target fusion features. During the training process, the model parameters of the initial point cloud segmentation model are continuously tuned. When the tuning results meet the preset conditions, the model training can be stopped to obtain the target point cloud segmentation model, so that the point cloud data can be segmented through the target point cloud segmentation model to obtain the corresponding instances. Therefore, in the process of model training, the local feature extraction network and the global feature extraction network are used for parallel processing, and the outputs of the two are fused to obtain the corresponding fusion features, thereby realizing the feature distinguishability of different instances in the learning point cloud data, improving the point cloud segmentation network's processing capabilities for complex scenes, reducing dependence on manually labeled data, improving data processing efficiency, and improving the accuracy and robustness of point cloud instance segmentation.

[0112] In some feasible implementations, during the model training process, the enhancement parameters for the target fusion features can be first determined, and then the target fusion features can be enhanced using the enhancement parameters to obtain enhanced fusion features. The enhanced fusion features are then used to train the initial point cloud segmentation model to generate a target point cloud segmentation model. Thus, by introducing a feature enhancement strategy, the model's ability to recognize instance boundaries can be effectively improved.

[0113] Among them, the enhancement parameters include at least the local density and geometric structure dynamics corresponding to the target point cloud data. The geometric structure dynamics can be obtained by analyzing the geometric properties of the point cloud data. The geometric properties can characterize the local shape and structural characteristics of the point cloud data, such as normal direction, curvature, local flatness, the number of points in the neighborhood, the volume of the neighborhood, etc. These properties can help the model better understand the characteristics of the point cloud data, so as to make more refined adjustments in the feature extraction process. According to the geometric properties, the learning strategy of the feature extraction network can be dynamically adjusted. For example, for areas with high curvature or low flatness, the learning rate can be increased or the weight decay can be reduced to better capture the characteristics of these complex areas.

[0114] In the specific implementation, the geometric structure can be used to dynamically determine the local density corresponding to the target point cloud data, and then the local density is used to enhance the target fusion feature to obtain the enhanced fusion feature. Among them, the local density can be calculated by the following formula:

[0115]

[0116] Among them, |N(p i )| is point p i The number of points in the neighborhood, V(p i ) is the volume of the neighborhood, ρ(p i ) is the local density. Further, it can be calculated by the following enhanced calculation formula:

[0117] f′ fused (p i )=Enhance(f(p i ),ρ(p i ))

[0118] Among them, f′ fused (p i ) is the enhanced fusion feature, f(p i ) is the target fusion feature, ρ(p i ) is the local density, and the Enhance function is based on the local density ρ(p i )The target fusion vector is enhanced, so that by introducing the feature enhancement strategy, the model’s ability to recognize instance boundaries can be effectively improved.

[0119] After obtaining the corresponding enhanced fusion features through the above process, the enhanced fusion features can be input into the initial point cloud segmentation model for prediction to obtain the prediction results corresponding to the enhanced fusion features, and the loss function between the prediction results and the preset true labels is calculated. Then, the loss function is used to tune the initial point cloud segmentation model until the loss function between the prediction results and the true labels meets the preset conditions, and the model training is stopped to obtain the target point cloud segmentation model. In the process of model training, the local feature extraction network and the global feature extraction network are used for parallel processing, and the outputs of the two are fused to obtain the corresponding fusion features, thereby realizing the feature distinguishability of different instances in the learning point cloud data, improving the point cloud segmentation network's processing capabilities for complex scenes, reducing the dependence on manually labeled data, improving data processing efficiency, and improving the accuracy and robustness of point cloud instance segmentation.

[0120] After obtaining the point cloud segmentation model through the above training process, the point cloud data to be segmented can be input into the point cloud segmentation model for instance segmentation. The point cloud segmentation model can use the corresponding clustering algorithm to divide the points in the point cloud data into different instances, thereby identifying the corresponding content of the point cloud data.

[0121] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0122] In an embodiment of the present invention, when performing instance segmentation on a point cloud, a corresponding point cloud segmentation model may be trained first, and then instance segmentation may be performed on the point cloud data based on the trained point cloud segmentation model. During the training process, the original point cloud data may be preprocessed first to obtain target point cloud data, and an initial point cloud segmentation model may be obtained. The initial point cloud segmentation model includes at least an initial global feature extraction network and an initial local feature extraction network. The target point cloud data is then input into the initial point cloud segmentation model, and a first global feature corresponding to the target point cloud data is extracted through the initial global feature extraction network in parallel, and a first local feature corresponding to the target point cloud data is extracted through the initial local feature extraction network. Then, the first local feature corresponding to the target point cloud data is extracted through the first local feature extraction network. A global feature is fused with the first local feature to obtain a target fusion feature, and finally the initial point cloud segmentation model is trained according to the target fusion feature to generate a target point cloud segmentation model. The target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances. Therefore, in the process of model training, the local feature extraction network and the global feature extraction network are used for parallel processing, and the outputs of the two are fused to obtain corresponding fusion features, which realizes the feature distinguishability of different instances in the learning point cloud data, improves the processing ability of the point cloud segmentation network for complex scenes, reduces the dependence on manually labeled data, improves data processing efficiency, and improves the accuracy and robustness of point cloud instance segmentation.

[0123] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0124] Reference Figure 2 , shows a schematic diagram of the model training process provided in an embodiment of the present invention, including: data preprocessing, self-supervised contrastive learning framework construction, adaptive feature enhancement, training and optimization, instance segmentation, post-processing and optimization, and model evaluation. Specifically:

[0125] 1. The original point cloud data can be preprocessed using a statistical removal method based on local density filtering to remove noise points by calculating the number of points in the neighborhood of each point and removing those points whose number of neighborhood points is less than a certain threshold. Then uniform downsampling or voxel downsampling is used to reduce the number of points in the point cloud and reduce computational complexity. Finally, the point cloud is spatially normalized to make the distribution of the point cloud data more uniform, which is helpful for network training.

[0126] 2. Reference Figure 3 , showing a schematic diagram of a self-supervised contrastive learning framework provided in an embodiment of the present invention, the original PointNet network and the local feature encoder are integrated in the following manner:

[0127] 1) A parallel structure is used to place the local feature encoder in parallel with the original feature extraction path of PointNet. For each point p in the point cloud i , the original PointNet network is responsible for extracting the global feature f global , while the local feature encoder is responsible for extracting the local feature f local (p i );

[0128] 2) Establish a feature fusion layer. After the maximum pooling layer of the PointNet network, add a feature fusion layer, which converts the global feature vector f global The feature vector f output by the local feature encoder local (p i ) to perform feature concatenation to form a new feature vector f fused (p i ):

[0129] f fused (p i )=[f global ;f local (p i )];

[0130] 3) Contrastive learning integration, the fused feature vector f fused (p i ) is input into the contrastive learning framework. By comparing and merging feature vectors during self-supervised training, the model learns features that distinguish the same instance from different instances;

[0131] 4) During the training process, the parameters of the global feature extraction network and the local feature encoder are compared through the contrast loss function L contrastive The optimization aims to maximize the feature similarity of points within the same instance and minimize the feature similarity of points between different instances. i , the contrast loss function can be expressed as:

[0132]

[0133] Among them, sim(.,.) is the similarity measurement function, N + is the positive sample set, N - is the set of negative samples.

[0134] 3. Reference Figure 4, shows a schematic diagram of the adaptive feature enhancement strategy provided in an embodiment of the present invention, the input layer performs local density estimation, and inputs the estimation result into the MLP layer, the output result of the MLP layer is input into the feature enhancement layer for feature enhancement strategy, and the enhancement result is input into the maximum pooling layer for further processing. Among them, the adaptive feature enhancement dynamically adjusts the learning strategy of the feature extraction network according to the local density and geometric structure of the point cloud data, and the specific method is:

[0135] 1) Perform local density estimation for each point in the point cloud. Assume that point p i The local density is ρ(p i ), calculated by the following formula:

[0136]

[0137] Among them, |N(p i )| is point p i The number of points in the neighborhood, V(p i ) is the volume of the neighborhood.

[0138] 2) Feature enhancement strategy: Adaptively adjust the learning strategy of the feature extraction network according to the estimated local density. Assume that the enhanced fusion feature vector is f' fused (p i ), then the feature enhancement calculation formula is:

[0139] f′ fused (p i )=Enhance(f(p i ),ρ(p i ))

[0140] Among them, the Enhance function is based on the local density ρ(p i ) to enhance the feature vector.

[0141] 4. Training and optimization: Use unlabeled point cloud data to train the self-supervised contrastive learning model and iteratively optimize the network parameters until the model converges.

[0142] 5. Instance segmentation: Use the trained model to extract features from point cloud data. The feature-based segmentation method divides the points in the point cloud into different instances.

[0143] 6. Post-processing and optimization,post-process the segmentation results, including removing small fragments, boundary optimization, etc., to improve the accuracy and practicality of instance segmentation.

[0144] 7. Model evaluation: evaluate the performance of the model through quantitative and qualitative evaluation methods to ensure its effectiveness and robustness in different scenarios.

[0145] Through the above process, during the model training process, the local feature extraction network and the global feature extraction network are used for parallel processing, and the outputs of the two are fused to obtain the corresponding fusion features, thereby realizing the feature distinguishability of different instances in the learning point cloud data, improving the point cloud segmentation network's processing capabilities for complex scenes, reducing dependence on manually labeled data, improving data processing efficiency, and improving the accuracy and robustness of point cloud instance segmentation.

[0146] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0147] Reference Figure 5 , shows a structural block diagram of a point cloud instance segmentation device provided in an embodiment of the present invention, which may specifically include the following modules:

[0148] The point cloud data processing module 501 is used to pre-process the original point cloud data to obtain target point cloud data;

[0149] A model acquisition module 502 is used to acquire an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network;

[0150] A feature extraction module 503 is used to input the target point cloud data into the initial point cloud segmentation model, and extract a first global feature corresponding to the target point cloud data through the initial global feature extraction network, and extract a first local feature corresponding to the target point cloud data through the initial local feature extraction network.

[0151] A feature fusion module 504 is used to fuse the first global feature with the first local feature to obtain a target fusion feature;

[0152] The model training module 505 is used to train the initial point cloud segmentation model according to the target fusion features to generate a target point cloud segmentation model, and the target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances.

[0153] In some feasible implementations, the feature fusion module 504 is specifically used to:

[0154] Concatenate the first global feature and the first local feature to obtain an initial fusion feature;

[0155] Inputting the initial fusion features into a preset learning model for self-supervised training to obtain a contrast loss function corresponding to the initial fusion model;

[0156] Based on the contrast loss function, the initial global feature extraction network and the initial local feature extraction network are parameter-tuned to obtain a target global feature extraction network and a target local feature extraction network;

[0157] The second global feature corresponding to the target point cloud data is extracted through the target global feature extraction network, and the second local feature corresponding to the target point cloud data is extracted through the target local feature extraction network, and the second global feature is fused with the second local feature to obtain a target fusion feature.

[0158] In some possible implementations, for each point p in the point cloud instance i , the contrast loss function is:

[0159]

[0160] Among them, f fused (p i ), f fused (p j ) are the initial fusion features, sim(.,.) is the similarity measurement function, N + is the positive sample set, N - is the set of negative samples, L contrastive (p i ) is the contrast loss function.

[0161] In some feasible implementations, the model training module 505 is specifically used to:

[0162] Determine enhancement parameters for the target fusion feature,

[0163] The target fusion feature is enhanced by using the enhancement parameter to obtain an enhanced fusion feature;

[0164] The enhanced fusion features are used to train the initial point cloud segmentation model to generate a target point cloud segmentation model.

[0165] In some feasible implementations, the enhancement parameters include at least the local density and geometric structure dynamics corresponding to the target point cloud data, and the model training module 505 is specifically used for:

[0166] Dynamically determining the local density corresponding to the target point cloud data using the geometric structure;

[0167] The target fusion feature is enhanced by using the local density to obtain an enhanced fusion feature.

[0168] In some feasible implementations, the model training module 505 is specifically used to perform calculations using the following formula:

[0169]

[0170] Among them, the |N(p i )| is point p i The number of points in the neighborhood, the V(p i ) is the volume of the neighborhood, and the ρ(p i ) is the local density.

[0171] In some feasible implementations, the model training module 505 is specifically used to perform calculations using the following formula:

[0172] f′ fused (p i )=Enhance(f(p i ),ρ(p i ))

[0173] Among them, the f′ fused (p i ) is an enhanced fusion feature, and the f(p i ) is the target fusion feature, and the ρ(p i ) is the local density, and the Enhance function is based on the local density ρ(p i ) to enhance the target fusion vector.

[0174] In some feasible implementations, the model training module 505 is specifically used to:

[0175] Inputting the enhanced fusion feature into the initial point cloud segmentation model for prediction, and obtaining a prediction result corresponding to the enhanced fusion feature;

[0176] Calculate the loss function between the prediction result and the preset true label;

[0177] The initial point cloud segmentation model is tuned using the loss function until the loss function between the predicted result and the true label meets the preset conditions, the model training is stopped, and the target point cloud segmentation model is obtained.

[0178] In some feasible implementations, the point cloud data processing module 201 is specifically used for:

[0179] Acquire original point cloud data, where the original point cloud data includes a plurality of points;

[0180] Obtaining the neighborhood corresponding to each of the points;

[0181] Calculate the number of other points in the neighborhood corresponding to each point in the original point cloud data based on local density filtering, and take the target point whose number of other points in the neighborhood is less than a preset threshold as a noise point, and remove the noise point from the original point cloud data;

[0182] For the original point cloud data with noise points removed, the original point cloud data is sampled by uniform downsampling or voxel downsampling, and the sampled original point cloud data is spatially normalized to obtain target point cloud data.

[0183] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned point cloud instance segmentation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0185] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned point cloud instance segmentation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0186] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0187] It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program codes.

[0188] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0192] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0193] The above is a detailed introduction to a point cloud instance segmentation method and a point cloud instance segmentation device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A point cloud instance segmentation method, characterized in that: include: Preprocess the original point cloud data to obtain target point cloud data; Acquire an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network; Inputting the target point cloud data into the initial point cloud segmentation model, extracting a first global feature corresponding to the target point cloud data through the initial global feature extraction network, and extracting a first local feature corresponding to the target point cloud data through the initial local feature extraction network in parallel; Fusing the first global feature with the first local feature to obtain a target fused feature; The initial point cloud segmentation model is trained according to the target fusion features to generate a target point cloud segmentation model, and the target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances.

2. The method according to claim 1, characterized in that: The fusing the global feature with the local feature to obtain a target fused feature includes: Concatenate the first global feature and the first local feature to obtain an initial fusion feature; Inputting the initial fusion features into a preset learning model for self-supervised training to obtain a contrast loss function corresponding to the initial fusion model; Based on the contrast loss function, the initial global feature extraction network and the initial local feature extraction network are parameter-tuned to obtain a target global feature extraction network and a target local feature extraction network; The second global feature corresponding to the target point cloud data is extracted through the target global feature extraction network, and the second local feature corresponding to the target point cloud data is extracted through the target local feature extraction network, and the second global feature is fused with the second local feature to obtain a target fused feature.

3. The method according to claim 2, characterized in that For each point p in the point cloud instance i , the contrast loss function is: Among them, f fused (p i ), f fused (p j ) are the initial fusion features, sim(.,.) is the similarity measurement function, N + is the positive sample set, N- is the negative sample set, L contrastive (p i ) is the contrast loss function.

4. The method according to claim 1 or 2, characterized in that: The training of the initial point cloud segmentation model according to the target fusion feature to generate a target point cloud segmentation model includes: Determine enhancement parameters for the target fusion feature, The target fusion feature is enhanced by using the enhancement parameter to obtain an enhanced fusion feature; The enhanced fusion features are used to train the initial point cloud segmentation model to generate a target point cloud segmentation model.

5. The method according to claim 4, characterized in that The enhancement parameters at least include the local density and geometric structure dynamics corresponding to the target point cloud data, and the use of the enhancement parameters to enhance the target fusion features to obtain the enhanced fusion features includes: Dynamically determining the local density corresponding to the target point cloud data using the geometric structure; The target fusion feature is enhanced by using the local density to obtain an enhanced fusion feature.

6. The method according to claim 5, characterized in that The local density corresponding to the target point cloud data is dynamically determined by using the geometric structure and is calculated by the following formula: Among them, the |N(p i )| is point p i The number of points in the neighborhood, the V(p i ) is the volume of the neighborhood, and the ρ(p i ) is the local density.

7. The method according to claim 5, characterized in that The target fusion feature is enhanced by using the local density to obtain the enhanced fusion feature, which is calculated by the following enhancement calculation formula: f′ fused (p i )=Enhance(f(p i ),ρ(p i )) Among them, the f′ fused (p i ) is an enhanced fusion feature, and the f(p i ) is the target fusion feature, and the ρ(p i ) is the local density, and the Enhance function is based on the local density ρ(p i ) to enhance the target fusion vector.

8. The method according to claim 1, characterized in that: The step of training the initial point cloud segmentation model using the enhanced fusion features to generate a target point cloud segmentation model includes: Inputting the enhanced fusion feature into the initial point cloud segmentation model for prediction, and obtaining a prediction result corresponding to the enhanced fusion feature; Calculate the loss function between the prediction result and the preset true label; The initial point cloud segmentation model is tuned using the loss function until the loss function between the predicted result and the true label meets the preset conditions, the model training is stopped, and the target point cloud segmentation model is obtained.

9. The method according to claim 1, characterized in that: The preprocessing of the original point cloud data to obtain the target point cloud data includes: Acquire original point cloud data, where the original point cloud data includes a plurality of points; Obtaining the neighborhood corresponding to each of the points; Calculate the number of other points in the neighborhood corresponding to each point in the original point cloud data based on local density filtering, and take the target point whose number of other points in the neighborhood is less than a preset threshold as a noise point, and remove the noise point from the original point cloud data; For the original point cloud data with noise points removed, the original point cloud data is sampled by uniform downsampling or voxel downsampling, and the sampled original point cloud data is spatially normalized to obtain target point cloud data.

10. A point cloud instance segmentation device, characterized in that: include: Point cloud data processing module, used to pre-process the original point cloud data to obtain target point cloud data; A model acquisition module, used to acquire an initial point cloud segmentation model, wherein the initial point cloud segmentation model at least includes an initial global feature extraction network and an initial local feature extraction network; A feature extraction module, used for inputting the target point cloud data into the initial point cloud segmentation model, and extracting a first global feature corresponding to the target point cloud data through the initial global feature extraction network, and extracting a first local feature corresponding to the target point cloud data through the initial local feature extraction network; A feature fusion module, used for fusing the first global feature with the first local feature to obtain a target fusion feature; The model training module is used to train the initial point cloud segmentation model according to the target fusion features to generate a target point cloud segmentation model, and the target point cloud segmentation model is used to segment the point cloud data to obtain corresponding instances.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 9 when executing the program stored in the memory.

12. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 9.

Images