Point cloud instance segmentation method and system based on synthetic data assistance

Abstract

The invention provides a synthetic data assistance-based point cloud instance segmentation method and system, and the method comprises the steps: generating an indoor 3D scene containing abundant objects through employing a three-dimensional 3D object model data set and 3D scene model data; on the basis, by counting a camera track when the scene point cloud is scanned, the track is simulated in the synthetic scene to carry out the same scanning process to obtain synthetic point cloud data with marks. In order to learn feature representation with rich semantics from the synthesized point cloud data, supervised pre-training is performed on the synthesized data, and a semantic segmentation task is used as an agent task of feature learning. According to the method, real scene data without labels are introduced in the pre-training process and are endowed with pseudo labels to guide the migration process from synthesis to real data. When the learned feature representation is used for a real point cloud instance segmentation task, data efficiency of real data can be realized through rich semantic knowledge and object shape knowledge contained in the pre-training model.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud instance segmentation in the field of computer vision, and in particular to a synthetic data-assisted point cloud instance segmentation method and system. Background Art

[0002] Semantic segmentation of point clouds requires knowing the semantic category of points. In addition to semantic categories, instance segmentation also requires knowing which points belong to the same object. There are multiple instances of a semantic category in the scene, such as multiple chairs in a room. The goal of point cloud instance segmentation is to find all objects in the point cloud and identify their semantic information, that is, to assign semantic labels and instance labels to each data point in the input point cloud. Semantic labels only require semantics (such as a point is a point on a table / chair). Instance segmentation requires that in addition to semantics, it is also necessary to distinguish which object the point belongs to. For example, if there are multiple chairs in the scene, instance segmentation requires that the semantics of a point is a chair and it is a point on the first chair. It is very difficult to understand the semantics corresponding to the geometric structure of a point cloud. On the one hand, point cloud data is unstructured and disordered. How to obtain effective representation from hundreds of thousands of point clouds is a problem that needs to be solved. On the other hand, the cost of collecting and annotating point cloud data is high, which leads to less data for training instance segmentation models. Practical applications require the use of a small amount of annotated data to achieve data-efficient instance segmentation. At present, some representative point cloud instance segmentation methods are:

[0003] 1) Point cloud instance segmentation based on sparse convolutional networks. Its main technical means are: using the U-Net network framework built with submanifold sparse convolution to extract features from the voxelized scene point cloud, and predicting the semantic category corresponding to each point and the center point of the object to which it belongs through these features. Then, semantic category prediction and center point prediction are used to cluster the point cloud in the point cloud space and the object center point voting space to obtain the clustered object. Problems and disadvantages: Training point cloud instance segmentation models requires dense semantic segmentation and instance segmentation annotations, which are costly in annotation. Unlike 2D models, which have large-scale data for pre-training, the feature representation of point clouds needs to be learned from scratch. Therefore, more data is required for supervised training to learn better point cloud feature representation, and data utilization is not efficient enough. This leads to a high cost for training models in practical applications.

[0004] 2) Point cloud instance segmentation method based on over-segmentation block clustering. Its main technical means is: the point cloud of the entire scene is represented as several over-segmentation blocks through a graph segmentation algorithm, and each over-segmentation block represents a geometrically uniform neighborhood. The point cloud instance segmentation task is transformed into learning a network to group over-segmentation blocks belonging to the same object instance and predict their semantics. The authors organized the over-segmentation blocks into a tree structure and performed hierarchical clustering based on this to obtain the final instance segmentation result. Problems and disadvantages: Representing the point cloud as an over-segmentation block requires the use of a grid representation. In many practical applications, there is only a point cloud but no 3D grid representation, which to some extent limits the use scenarios of this method. Due to the addition of over-segmentation operations, this also leads to a decrease in the real-time performance of the method.

[0005] 3) Data-efficient point cloud instance segmentation based on self-supervised feature representation learning. The main technical means is to design a comparative learning agent task through the matching relationship between the point clouds corresponding to two different frames of images in the same scene. The feature representation of the point cloud is learned in this self-supervised way, and it is used for feature initialization of downstream tasks such as instance segmentation. Problems and shortcomings: The learning of point cloud feature representation uses the matching relationship of points as supervision information, does not introduce semantic supervision information, and does not obtain high-level feature representation of the point cloud, which has limited effect in practical applications. In terms of data scale, the same batch of smaller data is still used, and the problem of scarce point cloud data is not solved. Summary of the invention

[0006] In order to solve the problems of scarcity of labeled point cloud data and inefficient data utilization in the above-mentioned prior art, the present invention proposes a method for assisting point cloud instance segmentation tasks by constructing synthetic point cloud data, so as to achieve data-efficient point cloud instance segmentation in practical applications.

[0007] The present invention generates a large number of indoor 3D scenes containing rich objects by using a large-scale, easily accessible, and diverse three-dimensional 3D object model data set and a small amount of 3D scene model data. On this basis, by counting the camera trajectory when scanning the scene point cloud, the trajectory is simulated in the synthetic scene to perform the same scanning process to obtain labeled synthetic point cloud data. In order to learn semantically rich feature representations from synthetic point cloud data, the present invention performs supervised pre-training on synthetic data and uses the semantic segmentation task as a proxy task for feature learning. At the same time, in order to reduce the knowledge transfer effect from synthetic data to real scene point cloud data, the present invention designs a domain-aware pre-training method. In the pre-training process, unlabeled real scene data is introduced and pseudo-labels are assigned to it to guide the migration process from synthesis to real data. When the learned feature representation is used for the real point cloud instance segmentation task, the data efficiency of the real data can be achieved through the rich semantic knowledge and object shape knowledge contained in the pre-training model. When only a small amount of annotations are used, the present invention can obtain more accurate instance segmentation results.

[0008] In order to obtain a good feature representation, a corresponding learning task is required and a loss function is constructed for training. In the method of the present invention, the task used for feature representation learning is different from the downstream task, so it is called a proxy task for feature representation learning.

[0009] Specifically, the present invention proposes a point cloud instance segmentation method based on synthetic data assistance, which includes:

[0010] The training data construction step includes obtaining three-dimensional scene data, collecting RGB-D data in the three-dimensional scene according to a preset camera trajectory, converting the RGB-D data into a point cloud, and using the point cloud in the world coordinate system as synthetic training data according to the camera trajectory, and each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; constructing a point cloud semantic segmentation model including a sparse convolutional network and a semantic segmentation task head;

[0011] In the pre-training step, the synthetic training data is input into the sparse convolutional network to obtain training features; the training features are semantically segmented by the semantic segmentation task head, the category to which each data point in the synthetic training data belongs is predicted, and the point cloud semantic segmentation model is updated by constructing cross entropy loss training with the category label as supervision information to obtain a pre-trained model;

[0012] In the model fine-tuning step, the parameters of the feature extraction network and / or semantic segmentation network in the pre-trained model are fine-tuned based on the real point cloud of the category to which the labeled data points belong as the real training data, so as to obtain the final point cloud instance segmentation model, so as to perform the cloud instance segmentation task on the target point cloud data and obtain the object category corresponding to each data point in the target point cloud data.

[0013] The synthetic data-assisted point cloud instance segmentation method, wherein the pre-training step further comprises:

[0014] The unlabeled real point cloud data is input into the point cloud semantic segmentation model, and the obtained semantic segmentation results are used as pseudo labels. The self-training loss is constructed by using the pseudo labels, and the total loss is constructed by combining the cross entropy loss to train and update the point cloud semantic segmentation model.

[0015] The total loss Where λ is a hyperparameter, The cross entropy loss for supervised synthetic data is is the self-training loss corresponding to the unlabeled real point cloud data;

[0016]

[0017]

[0018] in and are the category labels of the synthetic training data and the pseudo labels of the real point cloud data, SC i It is the semantic segmentation prediction result of the semantic segmentation task head.

[0019] The method for point cloud instance segmentation based on the assistance of synthetic data, wherein the pre-training step includes: generating the pseudo-label for all data of the unlabeled real point cloud data through the point cloud semantic segmentation model at the beginning of the point cloud data training; and regenerating the pseudo-label after each round of training update by the point cloud semantic segmentation model.

[0020] The synthetic data-assisted point cloud instance segmentation method, wherein the semantic segmentation task head adopts a multi-layer linear perceptron.

[0021] The present invention also proposes a point cloud instance segmentation system based on synthetic data assistance, which includes:

[0022] A training data construction module obtains three-dimensional scene data, collects RGB-D data in the three-dimensional scene according to a preset camera trajectory, converts the RGB-D data into a point cloud, and uses the point cloud in the world coordinate system as synthetic training data according to the camera trajectory, and each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; constructs a point cloud semantic segmentation model including a sparse convolutional network and a semantic segmentation task head;

[0023] The pre-training module inputs the synthetic training data into the sparse convolutional network to obtain training features; the semantic segmentation task head performs semantic segmentation on the training features, predicts the category to which each data point in the synthetic training data belongs, and uses the category label as supervision information to construct a cross entropy loss training to update the point cloud semantic segmentation model to obtain a pre-training model;

[0024] The model fine-tuning module fine-tunes the parameters of the feature extraction network and / or semantic segmentation network in the pre-trained model based on the real point cloud of the category to which the labeled data points belong as the real training data, and obtains the final point cloud instance segmentation model to perform the cloud instance segmentation task on the target point cloud data and obtain the object category corresponding to each data point in the target point cloud data.

[0025] The synthetic data-assisted point cloud instance segmentation system, wherein the pre-training module further includes:

[0026] The unlabeled real point cloud data is input into the point cloud semantic segmentation model, and the obtained semantic segmentation results are used as pseudo labels. The self-training loss is constructed by using the pseudo labels, and the total loss is constructed by combining the cross entropy loss to train and update the point cloud semantic segmentation model.

[0027] The total loss Where λ is a hyperparameter, The cross entropy loss for supervised synthetic data is is the self-training loss corresponding to the unlabeled real point cloud data;

[0028]

[0029]

[0030] in and are the category labels of the synthetic training data and the pseudo labels of the real point cloud data, SC i It is the semantic segmentation prediction result of the semantic segmentation task head.

[0031] The point cloud instance segmentation system based on synthetic data assistance, wherein the pre-training module includes: generating the pseudo-label for all data of the unlabeled real point cloud data through the point cloud semantic segmentation model at the beginning of the point cloud data training; and regenerating the pseudo-label once after each round of training update.

[0032] The synthetic data-assisted point cloud instance segmentation system, wherein the semantic segmentation task head adopts a multi-layer linear perceptron.

[0033] The present invention also proposes a storage medium for storing a program for executing the point cloud instance segmentation method.

[0034] The present invention also proposes a client for use in the point cloud instance segmentation system.

[0035] It can be seen from the above scheme that the advantages of the present invention are:

[0036] 1) Low demand for annotated point cloud data: This invention reduces the demand for annotated data for point cloud instance segmentation models by utilizing large-scale synthetic point cloud data to achieve data efficiency for downstream tasks. This can greatly reduce the model training and development costs for point cloud understanding tasks.

[0037] 2) Strong generalization and robustness: This paper adopts a realistic point cloud data construction process and considers the domain differences between synthetic data and downstream task data during model training. When migrating to downstream tasks, this method can be better adapted to the data of downstream tasks and has good generalization and robustness.

[0038] 3) High reusability: The pre-training method designed on synthetic point cloud data proposed in this paper can well learn the geometric structure and semantic information of point clouds and obtain feature representations that can be used for general point cloud understanding tasks. This representation is highly reusable and is not limited to specific application scenarios and point cloud understanding tasks.

[0039] 4) Flexible scalability: In practical applications, point cloud instance segmentation tasks often change in categories or application scenarios. The present invention can fully adapt to these changes and only requires a small amount of additional data annotation and slight modification of the network task header to be competent for new application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall process framework of the present invention;

[0041] Figure 2 A flowchart for constructing simulated synthetic point cloud data of the present invention;

[0042] Figure 3 This is a framework diagram of the synthetic data-assisted point cloud instance segmentation method of the present invention;

[0043] Figure 4 This is an example segmentation effect diagram of the present invention. DETAILED DESCRIPTION

[0044] In order to achieve the above technical effects, the present invention includes the following key technical points:

[0045] Key point 1: Use large-scale synthetic data for feature representation learning of point cloud tasks, which greatly reduces the task's demand for labeled real point cloud data. In point cloud instance segmentation tasks, a large amount of labeled data is required to support model training. However, in actual situations, the time and labor costs of collecting and annotating point cloud data are very high. The present invention can construct synthetic point cloud data in a low-cost manner and learn semantically rich feature representations from it. Based on this representation, accurate instance segmentation performance can be achieved using less real data annotation.

[0046] Key point 2, simulate the process of collecting point cloud data by humans, and construct realistic synthetic point cloud data in the synthetic scene. In the process of collecting point cloud data, different application scenarios will have different collection processes, and the collected point cloud data will also have certain differences in domain distribution. In order to make the synthetic data as close as possible to the real point cloud data and thus obtain better knowledge transfer effect, the present invention proposes to simulate the process and method of actually collecting point cloud data, and perform virtual collection in the synthetic scene. This method can minimize the domain difference between synthesis and reality and improve the migration effect of the model.

[0047] Key point 3, consider the domain differences between synthetic data and real data of downstream tasks in the pre-training stage. Since there are certain domain differences between synthetic data and data of downstream tasks, when the model pre-trained on synthetic data is migrated to downstream tasks, the migration effect will be not as expected due to the domain differences. In order to solve this problem, the present invention proposes a domain-aware pre-training method, which introduces unlabeled real data when performing supervised pre-training on synthetic data. This method can alleviate the problem of decreased migration performance caused by domain differences, and use feature representations with better migration performance to achieve data-efficient point cloud instance segmentation. The domain difference refers to the domain difference between synthetic and real data. It is mainly reflected in the differences in the density, geometry, completeness, edge smoothness, color distribution, etc. of the point cloud.

[0048] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0049] The technical route adopted by the present invention is (see Figure 1): Use a large-scale synthetic object dataset to build a synthetic scene, and construct annotated synthetic point cloud data by simulating the process of collecting data in the real scene. Based on the constructed large-scale synthetic data, a supervised learning task is constructed to guide the model to learn the geometric structure information and semantic information in the point cloud, so as to obtain efficient feature representation. In the process of feature representation learning, real unlabeled data is introduced to participate in the process to achieve feature learning that is known to the target domain, which is convenient for migration to downstream tasks. Finally, this feature representation is used as the model initialization for downstream real point cloud data training, so as to achieve instance segmentation model training that only requires a small amount of labeled real point cloud data.

[0050] Realistic synthetic point cloud data generation:

[0051] The construction process of synthetic point cloud data is as follows: Figure 2 As shown. Obtaining synthetic point cloud data mainly includes two parts: one is to construct a synthetic scene, and the other is to obtain point cloud data from the synthetic scene. There are many ways to generate synthetic scenes. For example, you can use cheap and easily available object CAD data (such as ShapeNet dataset) combined with 3D houses (such as SceneNet dataset) to generate synthetic scenes. In order to obtain realistic point cloud data, collecting point cloud data in synthetic scenes requires simulating the point cloud data collection process in real applications. Taking the indoor scene point cloud instance segmentation as an example, Figure 2 First, set up the virtual camera and make its camera parameters consistent with the RGB-D camera used in the real scene. Then count the distribution of the camera trajectory of the person in the process of collecting scene data, generate the camera trajectory in the synthetic scene and collect RGB-D data including RGB visual information and depth information. Convert the RGB-D data into a point cloud and combine it with the camera's position information to obtain the point cloud in the world coordinate system. After fusion and voxel downsampling, the final result is annotated, realistic synthetic point cloud data.

[0052] The process of converting RGB-D data into point cloud is to use RGB-D data and camera intrinsic parameters as input, and obtain the corresponding point cloud through formula transformation. The specific process is as follows: For a depth image pixel (u, v) and its corresponding depth value Z (u, v), assuming that the camera intrinsic parameters of the depth image are fx and fy (corresponding to the focal length, which is a constant), and the origin of the depth image is at the center of the image (u0, v0) (usually, it is half of the image width and height), then the corresponding point cloud coordinates (X, Y, Z) can be calculated using the following formula: X = (u-u0) * Z / fx; Y = (v-v0) * Z / fy; Z = Z (u, v), so that the coordinates (X, Y, Z) of each point in the three-dimensional point cloud are obtained.

[0053] The voxelization downsampling and voxelization downsampling include: since a single RGB-D image contains limited content and is small in size, multiple RGB-D images will be collected at multiple locations in a scene. The fusion here refers to placing the point clouds corresponding to multiple RGB-D images in the same coordinate system, so that they are spliced ​​and fused to form a larger point cloud. The input of this process is the point cloud corresponding to a single frame of RGB-D image and the camera position and posture when shooting the frame of data (i.e., the camera extrinsic parameter matrix). Voxelization downsampling mainly solves the problem that the point cloud after fusion is too dense in some locations.

[0054] Pre-training on synthetic data:

[0055] The learning of point cloud feature representation is achieved by performing supervised pre-training on synthetic point cloud data. In order to fully explore the semantic information and geometric structure information contained in the point cloud, the present invention uses the semantic segmentation task as a proxy task for the pre-training model. The method framework of this part is as follows Figure 3 As shown in the upper part a). Specifically, for the synthetic point cloud data, after voxelization, features are extracted through a U-Net network backbone composed of a sparse convolutional network. After the feature passes through the semantic segmentation task head composed of a multi-layer linear perceptron, the probability of the category to which each point belongs is predicted, and the semantic annotation of the synthetic point cloud is used as supervision information to guide the learning of the U-Net and semantic segmentation task head through cross entropy loss. Considering that there are domain differences between synthetic data and real point cloud data, in order to avoid the problem of decreased knowledge transfer performance due to domain differences, the present invention introduces unlabeled target domain real point cloud data in the pre-training process. The semantic segmentation results generated by the network for the unlabeled real data are used as pseudo labels. After self-training corresponding to the pseudo labels, the data distribution of the target domain data can be fully mined, and pre-training known to the target domain can be realized, reducing the difficulty of model migration. The role of the pseudo label is mainly to allow the network model to learn its geometric shape information on the unlabeled target domain data. Since this part of the data is not labeled, pseudo labels are generated by means of a network trained with synthetic data. First, a certain number of iterations are trained only on synthetic data. Then, unlabeled real data is introduced and trained simultaneously with synthetic data.

[0056] The loss corresponding to the self-training process can be expressed as the total loss Where λ is a hyperparameter, s is the set of object categories corresponding to the source domain, i.e., the synthetic point cloud data, and t is the set of object categories corresponding to the target domain, i.e., the real point cloud data. is the supervised loss for synthetic data, and the subscript “sem” is short for “semantic”. is the self-training loss corresponding to the unlabeled real point cloud data:

[0057]

[0058]

[0059] in and are the annotations of the synthetic data and the pseudo-labels of the i-th data point in the real data on the synthetic data task domain, SC i N is the semantic segmentation prediction probability of the network for the i-th data point. s is the total number of data points of synthetic training data, N t is the total number of data points of the real point cloud data. The probability output SC of the network and the pseudo-label Y are not generated at the same time. At the beginning of the real point cloud data training, a pseudo-label will be generated for all the data in the entire unlabeled real data set for the above training process. Whenever all the real data are trained once (that is, after one epoch of the real data), the present invention will regenerate a pseudo-label. It can be seen that the pseudo-label and probability output SC of the real point cloud are not generated at the same time.

[0060] Target domain real point instance segmentation training:

[0061] Using efficient feature representations learned through pre-training with synthetic data, such as Figure 3 As shown in Figure 2(b), the instance segmentation model can be trained in the target domain using a small amount of labeled data. Specifically, the backbone parameter part of the pre-trained model is used to initialize the network backbone of the downstream task model. Since the tasks in the synthetic domain and the segmentation tasks of real data in actual applications are usually different in category, we replace the task head of the synthetic domain model with a new task head. The model completes the training of the task head and fine-tuning of the network backbone model parameters through a small amount of labeled real point cloud data. Ultimately, a data-efficient point cloud instance segmentation task is achieved. When new point cloud data comes, accurate instance segmentation can be achieved through this model. Sample instance segmentation results of the model are as follows Figure 4 shown.

[0062] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0063] The present invention also proposes a point cloud instance segmentation system based on synthetic data assistance, which includes:

[0064] A training data construction module obtains three-dimensional scene data, collects RGB-D data in the three-dimensional scene according to a preset camera trajectory, converts the RGB-D data into a point cloud, and uses the point cloud in the world coordinate system as synthetic training data according to the camera trajectory, and each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; constructs a point cloud semantic segmentation model including a sparse convolutional network and a semantic segmentation task head;

[0065] The pre-training module inputs the synthetic training data into the sparse convolutional network to obtain training features; the semantic segmentation task head performs semantic segmentation on the training features, predicts the category to which each data point in the synthetic training data belongs, and uses the category label as supervision information to construct a cross entropy loss training to update the point cloud semantic segmentation model to obtain a pre-training model;

[0066] The model fine-tuning module fine-tunes the parameters of the feature extraction network and / or semantic segmentation network in the pre-trained model based on the real point cloud of the category to which the labeled data points belong as the real training data, and obtains the final point cloud instance segmentation model to perform the cloud instance segmentation task on the target point cloud data and obtain the object category corresponding to each data point in the target point cloud data.

[0067] The synthetic data-assisted point cloud instance segmentation system, wherein the pre-training module further includes:

[0068] The unlabeled real point cloud data is input into the point cloud semantic segmentation model, and the obtained semantic segmentation results are used as pseudo labels. The self-training loss is constructed by using the pseudo labels, and the total loss is constructed by combining the cross entropy loss to train and update the point cloud semantic segmentation model.

[0069] The total loss Where λ is a hyperparameter, The cross entropy loss for supervised synthetic data is is the self-training loss corresponding to the unlabeled real point cloud data;

[0070]

[0071]

[0072] in and are the category labels of the synthetic training data and the pseudo labels of the real point cloud data, SC i It is the semantic segmentation prediction result of the semantic segmentation task head.

[0073] The point cloud instance segmentation system based on synthetic data assistance, wherein the pre-training module includes: generating the pseudo-label for all data of the unlabeled real point cloud data through the point cloud semantic segmentation model at the beginning of the point cloud data training; and regenerating the pseudo-label once after each round of training update.

[0074] The synthetic data-assisted point cloud instance segmentation system, wherein the semantic segmentation task head adopts a multi-layer linear perceptron.

[0075] The present invention also proposes a storage medium for storing a program for executing the point cloud instance segmentation method.

[0076] The present invention also proposes a client for use in the point cloud instance segmentation system.

Claims

1. A point cloud instance segmentation method based on synthetic data, characterized in that: include: The training data construction step includes obtaining three-dimensional scene data, collecting RGB-D data in the three-dimensional scene according to a preset camera trajectory, converting the RGB-D data into a point cloud, and using the point cloud in the world coordinate system as synthetic training data according to the camera trajectory, and each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; constructing a point cloud semantic segmentation model including a sparse convolutional network and a semantic segmentation task head; In the pre-training step, the synthetic training data is input into the sparse convolutional network to obtain training features; the training features are semantically segmented by the semantic segmentation task head, the category to which each data point in the synthetic training data belongs is predicted, and the point cloud semantic segmentation model is updated by constructing cross entropy loss training with the category label as supervision information to obtain a pre-trained model; In the model fine-tuning step, the parameters of the feature extraction network and / or semantic segmentation network in the pre-trained model are fine-tuned based on the real point cloud of the category to which the labeled data points belong as the real training data, so as to obtain the final point cloud instance segmentation model, so as to perform the cloud instance segmentation task on the target point cloud data and obtain the object category corresponding to each data point in the target point cloud data.

2. The synthetic data-assisted point cloud instance segmentation method according to claim 1, characterized in that: The pre-training step also includes: The unlabeled real point cloud data is input into the point cloud semantic segmentation model, and the obtained semantic segmentation results are used as pseudo labels. The self-training loss is constructed by using the pseudo labels, and the total loss is constructed by combining the cross entropy loss to train and update the point cloud semantic segmentation model. The total loss Where λ is a hyperparameter, The cross entropy loss for supervised synthetic data is is the self-training loss corresponding to the unlabeled real point cloud data; in and are the category labels of the synthetic training data and the pseudo labels of the real point cloud data, SC i It is the semantic segmentation prediction result of the semantic segmentation task head.

3. The synthetic data-assisted point cloud instance segmentation method according to claim 2, characterized in that: The pre-training step includes: generating the pseudo-label for all data of the unlabeled real point cloud data through the point cloud semantic segmentation model at the beginning of the point cloud data training; and regenerating the pseudo-label for each round of training update by the point cloud semantic segmentation model.

4. The synthetic data-assisted point cloud instance segmentation method according to claim 1, characterized in that: The semantic segmentation task head uses a multi-layer linear perceptron.

5. A point cloud instance segmentation system based on synthetic data, characterized in that: include: A training data construction module obtains three-dimensional scene data, collects RGB-D data in the three-dimensional scene according to a preset camera trajectory, converts the RGB-D data into a point cloud, and uses the point cloud in the world coordinate system as synthetic training data according to the camera trajectory, and each data point in the synthetic training data corresponds to a category label of a corresponding object in the three-dimensional scene data; constructs a point cloud semantic segmentation model including a sparse convolutional network and a semantic segmentation task head; The pre-training module inputs the synthetic training data into the sparse convolutional network to obtain training features; the semantic segmentation task head performs semantic segmentation on the training features, predicts the category to which each data point in the synthetic training data belongs, and uses the category label as supervision information to construct a cross entropy loss training to update the point cloud semantic segmentation model to obtain a pre-training model; The model fine-tuning module fine-tunes the parameters of the feature extraction network and / or semantic segmentation network in the pre-trained model based on the real point cloud of the category to which the labeled data points belong as the real training data, and obtains the final point cloud instance segmentation model to perform the cloud instance segmentation task on the target point cloud data and obtain the object category corresponding to each data point in the target point cloud data.

6. The synthetic data-assisted point cloud instance segmentation system according to claim 5, characterized in that: The pre-training module also includes: The unlabeled real point cloud data is input into the point cloud semantic segmentation model, and the obtained semantic segmentation results are used as pseudo labels. The self-training loss is constructed by using the pseudo labels, and the total loss is constructed by combining the cross entropy loss to train and update the point cloud semantic segmentation model. The total loss Where λ is a hyperparameter, The cross entropy loss for supervised synthetic data is is the self-training loss corresponding to the unlabeled real point cloud data; in and are the category labels of the synthetic training data and the pseudo labels of the real point cloud data, SC i It is the semantic segmentation prediction result of the semantic segmentation task head.

7. The synthetic data-assisted point cloud instance segmentation system according to claim 6, characterized in that: The pre-training module includes: generating the pseudo-label for all data of the unlabeled real point cloud data through the point cloud semantic segmentation model at the beginning of point cloud data training; and regenerating the pseudo-label for each round of training update by the point cloud semantic segmentation model.

8. The synthetic data-assisted point cloud instance segmentation system according to claim 5, characterized in that: The semantic segmentation task head uses a multi-layer linear perceptron.

9. A storage medium for storing a program for executing any one of the point cloud instance segmentation methods as claimed in claims 1 to 4.

10. A client, used in any one of the point cloud instance segmentation systems of claims 5 to 8.

Images