Point cloud instance segmentation method and system based on synthetic data assistance

Abstract

The application provides a point cloud instance segmentation method and system based on synthetic data assistance, and the application generates an indoor 3D scene containing rich objects by using a 3D object model data set and a 3D scene model data. On this basis, the camera trajectory during scanning of the scene point cloud is counted, the same scanning process is simulated in the synthetic scene according to the trajectory to obtain labeled synthetic point cloud data. In order to learn semantic rich feature representation from the synthetic point cloud data, the application performs supervised pre-training on the synthetic data, and takes the semantic segmentation task as a proxy task for feature learning. In the pre-training process, the application introduces unlabeled real scene data and gives the data pseudo labels to guide the migration process from synthetic data to real data. When the learned feature representation is used for the real point cloud instance segmentation task, the rich semantic knowledge and object shape knowledge contained in the pre-training model can be used to realize data efficiency of the real data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud instance segmentation in computer vision, and particularly relates to a synthetic data assisted point cloud instance segmentation method and system. BACKGROUND

[0002] Semantic segmentation of point cloud needs to know the semantic category of points, and instance segmentation needs to know which points belong to the same object in addition to the semantic category. There are multiple instances of a semantic category in a scene, such as multiple chairs in a room. The goal of point cloud instance segmentation is to find all objects in the point cloud and identify their semantic information, that is, to assign a semantic label and an instance label to each data point in the input point cloud. The semantic label only requires to give the semantic (such as a point on a table / chair) and the instance segmentation requires to distinguish which object the point belongs to in addition to the semantic. For example, there are multiple chairs in the scene, and the instance segmentation requires to give a semantic of a chair and a point on the first chair. It is very difficult to understand the geometry corresponding to the semantics from the point cloud. On the one hand, point cloud data is non-structured and unordered, and how to obtain effective representation from hundreds of thousands of point clouds is a problem to be solved. On the other hand, the collection and annotation of point cloud data are costly, which leads to less data for training instance segmentation models. Actual application requires efficient instance segmentation of a small amount of labeled data. Some representative point cloud instance segmentation methods at present are as follows:

[0003] 1) Point cloud instance segmentation based on sparse convolutional network, the main technical means of which is to use a U-Net network framework built by sub-manifold sparse convolution to extract features of the scene point cloud after voxelization, and predict the semantic category corresponding to each point and the center point of the object to which the point belongs through the features. Then, the semantic category prediction and the center point prediction are used for point cloud clustering in the point cloud space and the object center point voting space, so as to obtain the clustered objects. Problems and shortcomings: training of point cloud instance segmentation model requires dense semantic segmentation and instance segmentation annotation, which is costly in annotation. Unlike 2D models, there are large-scale data for pre-training, and the feature representation of point cloud needs to be learned from scratch, so more data is needed for supervised training to learn better point cloud feature representation, and the data utilization is not efficient. This leads to a higher cost of training the model in actual application.

[0004] 2) A point cloud instance segmentation method based on oversegmentation block clustering. Its main technique involves representing the entire scene's point cloud as several oversegmented blocks using a graph segmentation algorithm. Each oversegmented block represents a geometrically uniform neighborhood. The point cloud instance segmentation task is transformed into learning a network to group oversegmented blocks belonging to the same object instance and predict their semantics. The authors organize the oversegmented blocks into a tree structure and perform hierarchical clustering based on this structure to obtain the final instance segmentation result. Problems and drawbacks: Representing the point cloud as oversegmented blocks requires a mesh representation. Many practical applications only have point clouds without a 3D mesh representation, which limits the application scenarios of this method to some extent. The added oversegmentation operation also reduces the method's real-time performance.

[0005] 3) Efficient point cloud instance segmentation based on self-supervised feature representation learning. The main technique involves designing a contrastive learning proxy task based on the matching relationship between point clouds corresponding to two different frames of the same scene. Feature representations of the point clouds are learned in this self-supervised manner and used for feature initialization in downstream tasks such as instance segmentation. Problems and drawbacks: The learning of point cloud feature representations uses the matching relationship of points as supervision information, without introducing semantic supervision information, and does not obtain high-level feature representations of the point clouds, resulting in limited effectiveness in practical applications. Furthermore, it still uses a relatively small batch of data, failing to address the problem of scarce point cloud data. Summary of the Invention

[0006] To address the problems of scarce labeled point cloud data and inefficient data utilization in the existing technologies, this invention proposes a method for assisting point cloud instance segmentation tasks by constructing synthetic point cloud data, thereby achieving efficient point cloud instance segmentation in practical applications.

[0007] The application generates a large number of indoor 3D scenes containing rich objects by using a large-scale, easily available and rich diversity of 3D object model data sets and a small amount of 3D scene model data. On this basis, the camera trajectory during the scanning of the scene point cloud is counted, and the same scanning process is simulated in the synthesized scene to obtain labeled synthesized point cloud data. In order to learn semantic rich feature representation from the synthesized point cloud data, the application performs supervised pre-training on the synthesized data, and takes the semantic segmentation task as a proxy task for feature learning. At the same time, in order to reduce the knowledge transfer effect from the synthesized data to the real scene point cloud data, the application designs a domain-aware pre-training method. In the pre-training process, unlabeled real scene data is introduced and pseudo labels are assigned to guide the migration process from the synthesized data to the real data. When the learned feature representation is used for real point cloud instance segmentation task, the rich semantic knowledge and object shape knowledge contained in the pre-training model can be used to realize data efficiency on real data. With only a small amount of labeling, the application can obtain more accurate instance segmentation results.

[0008] In order to obtain a good feature representation, a corresponding learning task is needed to construct a loss function for training. In the method of the application, the task for feature representation learning is not the same as the downstream task, so it is called a proxy task for feature representation learning.

[0009] Specifically, the application provides a point cloud instance segmentation method assisted by synthesized data, which comprises the following steps:

[0010] A training data construction step is provided, which acquires three-dimensional scene data, collects RGB-D data in the three-dimensional scene according to a preset camera trajectory, converts the RGB-D data into point cloud, and according to the camera trajectory, takes the point cloud in the world coordinate system as synthesized training data, wherein each data point in the synthesized training data corresponds to a category label of a corresponding object in the three-dimensional scene data; a point cloud semantic segmentation model comprising a sparse convolutional network and a semantic segmentation task head is constructed;

[0011] A pre-training step is provided, which inputs the synthesized training data into the sparse convolutional network to obtain training features; the training features are subjected to semantic segmentation through the semantic segmentation task head to predict the category to which each data point in the synthesized training data belongs, and the category label is taken as supervision information to train and update the point cloud semantic segmentation model by constructing a cross-entropy loss, thereby obtaining a pre-training model;

[0012] A model fine-tuning step is provided, which takes the real point cloud with labeled data points as real training data, fine-tunes the parameters of the feature extraction network and / or the semantic segmentation network in the pre-training model, and obtains a final point cloud instance segmentation model to perform a cloud instance segmentation task on target point cloud data, thereby obtaining the object category corresponding to each data point in the target point cloud data.

[0013] The point cloud instance segmentation method based on synthetic data assistance, wherein the pre-training step further comprises:

[0014] The unlabeled real point cloud data is input into the point cloud semantic segmentation model, and the obtained semantic segmentation result is used as a pseudo label, a self-training loss is constructed through the pseudo label, a total loss is constructed by combining the cross-entropy loss, and the point cloud semantic segmentation model is trained and updated;

[0015] The total loss Wherein λ is a hyperparameter, The cross-entropy loss supervised by synthetic data, The self-training loss corresponding to the unlabeled real point cloud data;

[0016]

[0017]

[0018] Wherein And The class label of the synthetic training data and the pseudo label of the real point cloud data are respectively SC i The semantic segmentation prediction result of the semantic segmentation task head.

[0019] The point cloud instance segmentation method based on synthetic data assistance, wherein the pre-training step comprises: generating the pseudo label for all data of the unlabeled real point cloud data by the point cloud semantic segmentation model when the point cloud data training starts; and regenerating the pseudo label once every time the point cloud semantic segmentation model is updated after one round of training.

[0020] The point cloud instance segmentation method based on synthetic data assistance, wherein the semantic segmentation task head adopts a multi-layer linear perception machine.

[0021] The application further provides a point cloud instance segmentation system based on synthetic data assistance, which comprises:

[0022] A training data construction module acquires three-dimensional scene data, collects RGB-D data in the three-dimensional scene according to a preset camera trajectory, converts the RGB-D data into point cloud, and takes the point cloud in a world coordinate system as synthetic training data according to the camera trajectory, wherein each data point in the synthetic training data corresponds to a class label of a corresponding object in the three-dimensional scene data; and a point cloud semantic segmentation model comprising a sparse convolutional network and a semantic segmentation task head is constructed.

[0023] a pre-training module, inputting the synthetic training data into the sparse convolutional network to obtain training features; performing semantic segmentation on the training features through the semantic segmentation task head to predict the category to which each data point in the synthetic training data belongs, and updating the point cloud semantic segmentation model through constructing a cross-entropy loss with the category label as supervision information to obtain a pre-trained model;

[0024] a model fine-tuning module, which fine-tunes parameters of the feature extraction network and / or the semantic segmentation network in the pre-trained model according to real point clouds of the labeled data points as real training data to obtain a final point cloud instance segmentation model to perform a cloud instance segmentation task on target point cloud data to obtain object categories corresponding to each data point in the target point cloud data.

[0025] The application further provides a point cloud instance segmentation system based on synthetic data assistance.

[0026] inputting unlabeled real point cloud data into the point cloud semantic segmentation model to obtain a semantic segmentation result as a pseudo label, constructing a self-training loss through the pseudo label, combining the cross-entropy loss and the self-training loss to construct a total loss, and training and updating the point cloud semantic segmentation model;

[0027] the total loss wherein λ is a hyperparameter, the cross-entropy loss supervised by the synthetic data, the self-training loss corresponding to the unlabeled real point cloud data;

[0028]

[0029]

[0030] wherein and are a category label of the synthetic training data and a pseudo label of the real point cloud data, respectively, SC i is a semantic segmentation prediction result of the semantic segmentation task head.

[0031] The application further provides a point cloud instance segmentation system based on synthetic data assistance.

[0032] The application further provides a point cloud instance segmentation system based on synthetic data assistance.

[0033] The application further provides a storage medium for storing a program for executing the point cloud instance segmentation method.

[0034] The application further provides a client for the point cloud instance segmentation system.

[0035] From the above scheme, the application has the following advantages:

[0036] 1) Low demand for labeled point cloud data: the application realizes data efficiency of downstream tasks by using large-scale synthetic point cloud data, reducing the demand for labeled data of the point cloud instance segmentation model. This can greatly reduce the model training and development cost of point cloud understanding tasks.

[0037] 2) Strong generalization ability and robustness: the application adopts a realistic point cloud data construction process, and considers the domain difference between synthetic data and downstream task data during model training. When migrating to downstream tasks, this method can better adapt to the data of downstream tasks, and has good generalization ability and robustness.

[0038] 3) Strong reusability: the pre-training method designed on synthetic point cloud data in the application can well learn the geometric structure and semantic information of point cloud, and obtain a feature representation that can be used for general point cloud understanding tasks. The representation has high reusability and is not limited to specific application scenarios and point cloud understanding tasks.

[0039] 4) Flexible scalability: in practical applications, point cloud instance segmentation tasks usually change in categories or application scenarios. The application can fully adapt to these changes, and only needs to provide a small amount of additional data labeling and slight network task head modification to meet new application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a schematic diagram of the overall process framework of the application;

[0041] Figure 2 is a flowchart of the realistic synthetic point cloud data construction process of the application;

[0042] Figure 3 is a framework diagram of the synthetic data assisted point cloud instance segmentation method of the application;

[0043] Figure 4 is an instance segmentation effect diagram of the application. DETAILED DESCRIPTION

[0044] In order to achieve the above technical effects, the application includes the following key technical points:

[0045] Key point 1, use large-scale synthetic data for feature representation learning of point cloud tasks, greatly reduce the demand for labeled real point cloud data for tasks. A large amount of labeled data is needed to support model training in the point cloud instance segmentation task. However, in actual situations, the time and labor cost of collecting and labeling point cloud data is very high. The present application can construct synthetic point cloud data in a low-cost manner and learn semantic-rich feature representations from it. Based on this representation, less real data labeling can achieve accurate instance segmentation performance.

[0046] Key point 2, simulate the process of collecting point cloud data by humans, and construct realistic synthetic point cloud data in synthetic scenes. Humans have different collection processes in different application scenarios when collecting point cloud data, and the point cloud data collected by them also has certain differences in domain distribution. In order to make the synthetic data as close to the real point cloud data as possible to obtain better knowledge transfer effect, the present application simulates the process and method of actually collecting point cloud data and virtually collects in synthetic scenes. This method can minimize the domain difference between synthetic and real data and improve the transfer effect of the model.

[0047] Key point 3, consider the domain difference between synthetic data and real data of downstream tasks in the pre-training stage. Since there is a certain domain difference between synthetic data and downstream task data, the model pre-trained on synthetic data will not perform as expected when it is transferred to downstream tasks due to domain difference. In order to solve this problem, the present application proposes a domain-aware pre-training method that introduces unlabeled real data during supervised pre-training on synthetic data. This method can alleviate the performance decline problem caused by domain difference, and use the feature representation with better transfer performance to achieve efficient point cloud instance segmentation. The domain difference refers to the domain difference between synthetic and real data. It mainly reflects the differences in point cloud density, geometric shape, completeness, edge smoothness, color distribution, etc.

[0048] In order to make the above features and effects of the present application more clear and easy to understand, the following embodiments are described in detail below, and the accompanying drawings are described as follows.

[0049] The technical route adopted by the present application is (see Figure 1): synthetic scenes are constructed using large-scale synthetic object datasets, and annotated synthetic point cloud data is constructed by simulating the process of collecting data in real scenes. On the basis of the large-scale synthetic data constructed, supervised learning tasks are constructed to guide the model to learn the geometric structure information and semantic information in the point cloud, so as to obtain efficient feature representation. In the feature representation learning process, real unlabeled data is introduced to participate in the process, realizing target domain aware feature learning, and facilitating migration to downstream tasks. Finally, the feature representation is used as the initialization of the model trained on downstream real point cloud data, realizing the training of the instance segmentation model only requiring a small amount of labeled real point cloud data.

[0050] Synthetic point cloud data generation:

[0051] The construction process of synthetic point cloud data is shown in Figure 2 . Obtaining synthetic point cloud data mainly includes two parts, one is to construct a synthetic scene, and the other is to obtain point cloud data from the synthetic scene. There are many ways to generate a synthetic scene, such as using cheap and easily available object CAD data (such as the ShapeNet dataset) combined with 3D houses (such as the SceneNet dataset) to generate a synthetic scene. In order to obtain realistic point cloud data, the collection process of point cloud data in real applications needs to be simulated in the synthetic scene. Taking indoor scene point cloud instance segmentation as an example, Figure 2 , first set the virtual camera and make its camera parameters consistent with the RGB-D camera used in the real scene. Then, the camera trajectory distribution of people in the process of collecting scene data is counted, and the camera trajectory is generated in the synthetic scene to collect RGB-D data including RGB visual information and depth information. The RGB-D data is converted to point cloud and combined with the position information of the camera to obtain the point cloud in the world coordinate system. After fusion and voxelization downsampling, the final labeled and realistic synthetic point cloud data is obtained.

[0052] The process of converting RGB-D data to point cloud is to take RGB-D data and camera intrinsic parameters as input, and obtain the corresponding point cloud through formula transformation. The specific process is as follows: for a depth image pixel point (u, v) and its corresponding depth value Z(u, v), assuming that the camera intrinsic parameters of the depth image are fx and fy (corresponding to the focal length, which is a constant), and the origin of the depth image is at the center of the image (u0, v0) (usually, it is half of the image width and height), then the corresponding point cloud coordinates (X, Y, Z) can be calculated using the following formula: X = (u-u0)*Z / fx; Y = (v-v0)*Z / fy; Z = Z(u, v), so that the coordinates (X, Y, Z) of each point in the three-dimensional point cloud are obtained.

[0053] The voxelization downsampling and the voxelization downsampling include: due to the limited content contained in a single RGB-D image, a plurality of RGB-D images are collected at a plurality of positions in a scene, and the fusion here refers to placing the point clouds corresponding to the plurality of RGB-D images in the same coordinate system to make the point clouds spliced and fused to form a point cloud with a larger size. The input of this process is the point cloud corresponding to a single frame of RGB-D image and the camera position and attitude (i.e. camera extrinsic matrix) when the frame of data is shot. The voxelization downsampling mainly solves the problem that the point cloud after fusion is too dense at some positions.

[0054] Synthetic data pre-training:

[0055] Point cloud feature representation learning is achieved by supervised pre-training on synthetic point cloud data. In order to fully exploit the semantic information and geometric structure information contained in the point cloud, the present application adopts the semantic segmentation task as the proxy task of the pre-training model. The method framework of this part is shown in the upper half of Fig. a). Figure 3 Specifically, for synthetic point cloud data, the features are extracted by a U-Net network backbone composed of a sparse convolutional network after voxelization. After the features pass through the semantic segmentation task head composed of multiple linear perception structures, the probability of each point belonging to a class is predicted, and the semantic annotation of the synthetic point cloud is used as the supervision information to guide the learning of the U-Net and the semantic segmentation task head through the cross-entropy loss. Considering the domain difference between synthetic data and real point cloud data, in order to avoid the problem of decreased knowledge transfer performance due to domain difference, the present application introduces unlabeled target domain real point cloud data in the pre-training process. The semantic segmentation result generated by the network is used as a pseudo label for the unlabeled real data. After self-training corresponding to the pseudo label, the data distribution of the target domain data can be fully exploited, the pre-training of the target domain is realized, and the difficulty of model migration is reduced. The main role of the pseudo label is to let the network model learn the geometric shape information on the unlabeled target domain data. Since this part of data is not labeled, the network trained on synthetic data is used to generate pseudo labels. First, train for a certain number of iterations only on synthetic data. Then introduce unlabeled real data and train simultaneously with synthetic data.

[0056] The loss corresponding to the self-training process can be expressed as the total loss Where λ is a hyperparameter, s is the object class set corresponding to the source domain, i.e. synthetic point cloud data, t is the object class set corresponding to the target domain, i.e. real point cloud data, is the supervised loss of synthetic data, and the subscript "sem" is the abbreviation of "semantic", is the self-training loss corresponding to the unlabeled real point cloud data:

[0057]

[0058]

[0059] wherein and are the label of synthetic data and the pseudo label of the i-th data point in real data in the synthetic data task domain respectively, SC i is the semantic segmentation prediction probability of the i-th data point by the network. N s is the total number of data points of the synthetic training data, N t is the total number of data points of the real point cloud data. The probability output SC and the pseudo label Y of the network are not generated at the same time. At the beginning of the real point cloud data training, the pseudo label of all data in the entire unlabeled real data set is generated once, which is used in the above training process. Each time all the real data is trained once (i.e. after an epoch of real data), the pseudo label is regenerated once. It can be seen that the pseudo label of the real point cloud and the probability output SC are not generated at the same time.

[0060] Target domain real point instance segmentation training:

[0061] Using the efficient feature representation learned by the synthetic data pre-training, such as Figure 3 b) shown, the training of the instance segmentation model can be completed on the target domain using a small amount of labeled data. Specifically, the backbone parameter part of the pre-trained model is used to initialize the network backbone of the downstream task model. Since the task of the synthetic domain and the segmentation task of the real data in the actual application are usually different in class, we replace the task head of the synthetic domain model with a new task head. The model completes the training of the task head and the fine-tuning of the network backbone model parameter through a small amount of labeled real point cloud data. Finally, the efficient point cloud instance segmentation task of data is realized. When new point cloud data comes, accurate instance segmentation can be realized through the model. The instance segmentation result sample of the model is shown in Figure 4 .

[0062] The following is a system embodiment corresponding to the above method embodiment. The present embodiment can be implemented in cooperation with the above-mentioned embodiments. The related technical details mentioned in the above-mentioned embodiments are still valid in the present embodiment. In order to reduce repetition, they will not be repeated here. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the above-mentioned embodiments.

[0063] The present application also proposes a point cloud instance segmentation system based on synthetic data assistance, which comprises:

[0064] The training data construction module obtains three-dimensional scene data, collects RGB-D data in the three-dimensional scene according to a preset camera trajectory, converts the RGB-D data into a point cloud, and according to the camera trajectory, takes the point cloud in a world coordinate system as synthetic training data, and each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; a point cloud semantic segmentation model including a sparse convolutional network and a semantic segmentation task head is constructed;

[0065] The pre-training module inputs the synthetic training data into the sparse convolutional network to obtain training features; the training features are subjected to semantic segmentation through the semantic segmentation task head to predict the category to which each data point in the synthetic training data belongs, and the point cloud semantic segmentation model is trained and updated by constructing a cross-entropy loss with the category label as supervision information to obtain a pre-trained model.

[0066] The model fine-tuning module takes a real point cloud of a category of labeled data points as real training data to fine-tune parameters of a feature extraction network and / or a semantic segmentation network in the pre-trained model to obtain a final point cloud instance segmentation model to perform a cloud instance segmentation task on target point cloud data to obtain object categories corresponding to data points in the target point cloud data.

[0067] The point cloud instance segmentation system based on synthetic data assisted, wherein the pre-training module further comprises:

[0068] The unlabeled real point cloud data is input into the point cloud semantic segmentation model to obtain a semantic segmentation result as a pseudo label, a self-training loss is constructed through the pseudo label, a total loss is constructed in combination with the cross-entropy loss, and the point cloud semantic segmentation model is trained and updated;

[0069] The total loss wherein λ is a hyperparameter, the cross-entropy loss supervised by the synthetic data, the self-training loss corresponding to the unlabeled real point cloud data;

[0070]

[0071]

[0072] wherein and are a category label of the synthetic training data and a pseudo label of the real point cloud data, respectively, and SC i is a semantic segmentation prediction result of the semantic segmentation task head.

[0073] The point cloud instance segmentation system based on synthetic data assistance, wherein the pre-training module comprises: generating the pseudo label for all data of the unlabeled real point cloud data by the point cloud semantic segmentation model at the beginning of point cloud data training; and the point cloud semantic segmentation model regenerates the pseudo label once after each training update.

[0074] The point cloud instance segmentation system based on synthetic data assistance, wherein the semantic segmentation task head adopts a multi-layer linear perception machine.

[0075] The application further provides a storage medium for storing a program for executing the point cloud instance segmentation method.

[0076] The application further provides a client for the point cloud instance segmentation system.

Claims

1. A method for point cloud instance segmentation based on synthetic data assistance, characterized in that, The method comprises the following steps: A training data construction step is performed to obtain three-dimensional scene data, RGB-D data is collected in the three-dimensional scene according to a preset camera trajectory, the RGB-D data is converted into point cloud, and the point cloud in a world coordinate system is taken as synthetic training data according to the camera trajectory, wherein each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; a point cloud semantic segmentation model comprising a sparse convolutional network and a semantic segmentation task head is constructed; A pre-training step is performed to input the synthetic training data into the sparse convolutional network to obtain training features; the training features are subjected to semantic segmentation through the semantic segmentation task head to predict the category to which each data point in the synthetic training data belongs, and the point cloud semantic segmentation model is trained and updated by constructing a cross-entropy loss with the category label as supervision information to obtain a pre-training model; A model fine-tuning step is performed to take a real point cloud with labeled data points as real training data to fine-tune parameters of a feature extraction network and / or a semantic segmentation network in the pre-training model to obtain a final point cloud instance segmentation model, so as to perform a cloud instance segmentation task on target point cloud data to obtain object categories corresponding to data points in the target point cloud data; The pre-training step further comprises the following steps: Unlabeled real point cloud data is input into the point cloud semantic segmentation model to obtain a semantic segmentation result as a pseudo label, a self-training loss is constructed by using the pseudo label, a total loss is constructed by combining the cross-entropy loss and the self-training loss, and the point cloud semantic segmentation model is trained and updated; the total loss wherein is a hyperparameter, is the cross-entropy loss for synthetic data supervised, is a self-training loss corresponding to the unlabelled real point cloud data; = wherein and are a class label of the synthetic training data and a pseudo label of the real point cloud data, respectively, is a semantic segmentation prediction result of the semantic segmentation task head; is a total number of data points of the synthetic training data, is a total number of data points of the real point cloud data.

2. The synthetic data aided based point cloud instance segmentation method of claim 1, wherein, The pre-training step comprises the following steps: the pseudo label is generated for all data of the unlabeled real point cloud data by using the point cloud semantic segmentation model at the beginning of point cloud data training; the pseudo label is regenerated by the point cloud semantic segmentation model once after each round of training and updating.

3. The synthetic data aided based point cloud instance segmentation method of claim 1, wherein, The semantic segmentation task head adopts a multi-layer linear perception machine.

4. A synthetic data aided based point cloud instance segmentation system, characterized in that, The method comprises the following steps: A training data construction module is configured to obtain three-dimensional scene data, RGB-D data is collected in the three-dimensional scene according to a preset camera trajectory, the RGB-D data is converted into point cloud, and the point cloud in a world coordinate system is taken as synthetic training data according to the camera trajectory, wherein each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; a point cloud semantic segmentation model comprising a sparse convolutional network and a semantic segmentation task head is constructed; A pre-training module is configured to input the synthetic training data into the sparse convolutional network to obtain training features; the training features are subjected to semantic segmentation through the semantic segmentation task head to predict the category to which each data point in the synthetic training data belongs, and the point cloud semantic segmentation model is trained and updated by constructing a cross-entropy loss with the category label as supervision information to obtain a pre-training model; A model fine-tuning module is configured to take a real point cloud with labeled data points as real training data to fine-tune parameters of a feature extraction network and / or a semantic segmentation network in the pre-training model to obtain a final point cloud instance segmentation model, so as to perform a cloud instance segmentation task on target point cloud data to obtain object categories corresponding to data points in the target point cloud data; The pre-training module further comprises the following steps: The unlabeled real point cloud data is input into the point cloud semantic segmentation model, a semantic segmentation result obtained is used as a pseudo label, a self-training loss is constructed through the pseudo label, a total loss is constructed in combination with the cross entropy loss, and the point cloud semantic segmentation model is trained and updated; the total loss wherein is an over-parameter, is the cross-entropy loss of the synthetic data supervised, is the self-training loss corresponding to the unlabeled real point cloud data; = wherein and are a class label of the synthetic training data and a pseudo label of the real point cloud data, respectively, is a semantic segmentation prediction result of the semantic segmentation task head; is a total number of data points of the synthetic training data, is a total number of data points of the real point cloud data.

5. The synthetic data aided based point cloud instance segmentation system of claim 4, wherein, The pre-training module includes: generating the pseudo label for all data of the unlabeled real point cloud data through the point cloud semantic segmentation model when the point cloud data training starts; and regenerating the pseudo label once every time the point cloud semantic segmentation model is updated after one round of training.

6. The synthetic data aided based point cloud instance segmentation system of claim 4, wherein, The semantic segmentation task head adopts a multi-layer linear perception machine. 7.A storage medium for storing a program for executing the point cloud instance segmentation method according to any one of claims 1 to 3. 8.A client for the point cloud instance segmentation system according to any one of claims 4 to 6.

Images