Example segmentation method, apparatus, device, storage medium, and program product

Abstract

An instance segmentation method, device, equipment, storage medium and program product are provided. The method comprises: performing semantic prediction and centroid offset prediction on a to-be-segmented point cloud through a first backbone network to obtain a semantic prediction result and a centroid offset vector of each point; performing multi-level object prediction on the to-be-segmented point cloud through a second backbone network to obtain a multi-level point cloud; moving each point of the multi-level point cloud to a predicted center according to the centroid offset vector to obtain an influence point cloud; clustering core points of the influence point cloud according to the semantic prediction result to obtain at least one core point cloud of each semantic category; and clustering edge points in the influence point cloud into target core point clouds of corresponding semantic categories to obtain an instance segmentation result.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, computer vision technology, and in particular to an instance segmentation method, apparatus, device, storage medium, and program product. Background Technology

[0002] Currently, point cloud instance segmentation methods based on deep learning are usually strongly supervised methods, meaning that a large number of training samples need to be labeled with semantic and instance labels. However, the large amount of labeling work increases the manual cost of instance segmentation and affects its efficiency. On the other hand, weakly supervised methods can reduce the amount of labeling and the manual cost, which improves the efficiency of instance segmentation to some extent, but affects the accuracy of the instance segmentation results. Summary of the Invention

[0003] This disclosure provides at least one instance segmentation method, apparatus, device, storage medium, and program product, which improves the efficiency and accuracy of instance segmentation.

[0004] The technical solution disclosed herein is implemented as follows:

[0005] On one hand, embodiments of this disclosure provide an instance segmentation method, the method comprising:

[0006] The first backbone network performs semantic prediction and centroid offset prediction on the acquired point cloud to be segmented, obtaining the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels; the second backbone network performs object-orientation prediction on the point cloud to be segmented, obtaining a multi-level point cloud including multiple levels of object-orientation labels; the multiple levels of object-orientation labels are used to characterize the shape of each instance; according to the centroid offset vector, each point in the multi-level point cloud is moved to its corresponding prediction center to obtain an influence point cloud; according to the semantic prediction result, the core points in the influence point cloud are clustered to obtain at least one core point cloud for each semantic category; the core points are points whose object-orientation label level is a preset core level; the edge points in the influence point cloud are clustered into the target core point cloud in at least one core point cloud of the corresponding semantic category to obtain the instance segmentation result; the edge points are points whose object-orientation label level is a preset edge level.

[0007] In this way, the training of the first and second backbone networks in the instance segmentation device does not use pre-labeled instance labels, which reduces the workload of instance labeling, improves network training efficiency, and thus improves the efficiency of instance segmentation. Furthermore, since the instance segmentation device processes core points located in the core region of the instance and edge points located in the edge region separately, it improves the accuracy of edge point clustering, thereby improving the accuracy of instance segmentation.

[0008] On the other hand, embodiments of this disclosure provide an instance segmentation apparatus, including:

[0009] The first prediction module is used to perform semantic prediction and centroid offset prediction on the acquired point cloud to be segmented through the first backbone network, so as to obtain the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels.

[0010] The second prediction module is used to perform object-oriented prediction on the point cloud to be segmented through the second backbone network to obtain a multi-level point cloud including multiple levels of object-oriented labels; the multiple levels of object-oriented labels are used to characterize the shape of each instance;

[0011] The moving module is used to move each point in the multi-level point cloud to its corresponding prediction center according to the centroid offset vector, so as to obtain the influence point cloud;

[0012] The core clustering module is used to cluster the core points in the influence point cloud according to the semantic prediction results, so as to obtain at least one core point cloud for each semantic category; the core points are points whose object label level is a preset core level.

[0013] The edge clustering module is used to cluster edge points in the influence point cloud into the target core point cloud in at least one core point cloud of the corresponding semantic category to obtain the instance segmentation result; the edge points are points whose object label level is a preset edge level.

[0014] In another aspect, embodiments of this disclosure provide an instance segmentation device, including:

[0015] Memory, used to store computer programs;

[0016] The processor, when executing the computer program stored in the memory, implements the instance segmentation method described above.

[0017] In another aspect, embodiments of this disclosure provide a computer-readable storage medium storing a computer program for implementing the above-described instance segmentation method when executed by a processor.

[0018] In another aspect, embodiments of this disclosure provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, the above-described instance segmentation method is implemented.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0021] Figure 1 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0022] Figure 2 This is a schematic diagram illustrating the effect of centroid regression provided in an embodiment of this disclosure;

[0023] Figure 3 A schematic diagram of an optional multi-level predicted point cloud provided for an embodiment of this disclosure;

[0024] Figure 4 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0025] Figure 5 An optional core point influence map provided for an embodiment of this disclosure;

[0026] Figure 6A A schematic diagram of an optional clustering method in a related art provided in this disclosure embodiment;

[0027] Figure 6B A schematic diagram illustrating an optional clustering effect provided for an embodiment of this disclosure;

[0028] Figure 7 A schematic diagram of an optional instance segmentation method provided in an embodiment of this disclosure;

[0029] Figure 8 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0030] Figure 9 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0031] Figure 10 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0032] Figure 11A A schematic diagram of an optional multi-level training point cloud provided for an embodiment of this disclosure;

[0033] Figure 11B A schematic diagram of an optional recombined training point cloud provided in an embodiment of this disclosure;

[0034] Figure 11C A schematic diagram of an optional reconfigurable virtual scene provided for an embodiment of this disclosure;

[0035] Figure 12 A schematic diagram of an optional recombination template provided for an embodiment of this disclosure;

[0036] Figure 13 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0037] Figure 14 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0038] Figure 15 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0039] Figure 16 A flowchart of an optional instance segmentation method provided for an embodiment of this disclosure;

[0040] Figure 17 A schematic diagram illustrating the training process of an optional instance segmentation network provided in an embodiment of this disclosure;

[0041] Figure 18A An optional initial pseudo-instance point cloud diagram provided for an embodiment of this disclosure;

[0042] Figure 18B A schematic diagram of an optional filtered first training point cloud provided for an embodiment of this disclosure;

[0043] Figure 18C An optional instance segmentation effect diagram provided for an embodiment of this disclosure;

[0044] Figure 18D An optional multi-level predicted point cloud diagram provided for an embodiment of this disclosure;

[0045] Figure 18E An optional core point clustering diagram provided for an embodiment of this disclosure;

[0046] Figure 18F A schematic diagram illustrating an optional instance segmentation effect provided in an embodiment of this disclosure;

[0047] Figure 19A An optional multi-level predicted point cloud diagram provided for an embodiment of this disclosure;

[0048] Figure 19B A schematic diagram illustrating an optional core point clustering effect provided in this embodiment of the disclosure;

[0049] Figure 19C A schematic diagram illustrating an optional core point clustering effect provided in this embodiment of the disclosure;

[0050] Figure 20 A schematic diagram of an optional instance segmentation device provided in an embodiment of this disclosure;

[0051] Figure 21 A schematic diagram of the hardware entity of an optional instance segmentation device provided in an embodiment of this disclosure. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0053] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0055] To better understand the instance segmentation method provided in the embodiments of this disclosure, the instance segmentation schemes used in related technologies will be described below.

[0056] With the rapid development of 3D acquisition technology, high-performance 3D scanners such as LiDAR and RGB-D depth cameras have greatly reduced the difficulty of acquiring point cloud data, while also bringing about more and more emerging applications that require perception of the 3D environment and object interaction. As a crucial and fundamental perception task in 3D scene understanding, 3D instance segmentation requires simultaneously assigning specific semantic labels and instance labels to each point in the point cloud to distinguish different individuals. It is widely used in real-world scenarios such as indoor robot navigation, augmented reality, and autonomous driving.

[0057] Currently, most 3D instance segmentation methods are strongly supervised methods. For example, they first perform object detection to obtain 3D bounding boxes, and then remove points with low instance confidence from the 3D bounding boxes; alternatively, they may not explicitly obtain 3D bounding boxes, but instead group feature points, etc. Strongly supervised methods all require joint prediction of semantics and instances; however, the annotation of instance labels requires a significant amount of manual time.

[0058] For example, in a scenario, there are 20 chairs that share the same semantic meaning—chairs—which can be labeled at once; however, instance labeling requires labeling each chair individually, which takes more labeling time.

[0059] Later, weakly supervised methods for 3D instance segmentation were proposed, which reduced the number of point clouds that needed to be labeled; for example, only 3D bounding boxes were labeled or only one point was labeled for each instance; however, the performance of instance segmentation networks trained with such labeled point clouds was far behind that of strongly supervised methods.

[0060] This disclosure provides an instance segmentation method, which can be executed by the processor of an instance segmentation device. The instance segmentation device refers to a device with instance segmentation capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 This is a schematic diagram illustrating the implementation process of an instance segmentation method provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the method includes the following steps: S101-S105.

[0061] S101. Semantic prediction and centroid offset prediction are performed on the acquired point cloud to be segmented through the first backbone network to obtain the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels.

[0062] In this embodiment of the disclosure, the point cloud to be segmented is a 3D point cloud acquired from a scene using an image acquisition device or a 3D scanner, where each point in the 3D point cloud resides in a three-dimensional coordinate system. The method for acquiring the 3D point cloud from the scene can be set as needed, and this embodiment of the disclosure does not impose any limitations.

[0063] In this embodiment of the disclosure, the first backbone network can perform semantic prediction and centroid offset prediction on the point cloud to be segmented to obtain a semantic prediction result and a centroid offset prediction vector; wherein, the semantic prediction result is a point cloud to be segmented with the semantic label of each predicted point; the centroid offset vector represents the offset vector between the predicted instance centroid and the points in the corresponding instance.

[0064] In this embodiment of the disclosure, the instance segmentation device can train an initial first backbone network based on a first training point cloud to obtain a first backbone network; the first training point cloud contains pseudo-instance labels, and the instance centroids of the pseudo-instances in the first training point cloud can be used as the centroids of the real instances; in this way, the instance segmentation device can predict the centroid offset vector through learning.

[0065] In this embodiment, the second backbone network is also trained based on the first training point cloud. The instance segmentation device can annotate multiple levels of object labels based on the first training point cloud to obtain a multi-level training point cloud. The multi-level training point cloud includes multiple levels of object labels, which are used to characterize the shape of each instance in the point cloud to be segmented. The object level label of each point in the multi-level training point cloud is used to characterize whether the point belongs to the core region or the edge region of the corresponding instance. The initial second backbone network is trained using the multi-level training point cloud to obtain the second backbone network.

[0066] It should be noted that the number of multiple levels can be set as needed, and this embodiment of the disclosure does not limit this. Multiple levels include core levels and edge levels.

[0067] In the embodiments of this disclosure, the first backbone network and the second backbone network can be the same network or different networks; here, the first backbone network or the second backbone network can be a convolutional neural network, such as: 3D U-net based on submanifold sparse convolution, Point-Voxel CNN (PVCNN), Persistent Homology based Graph Convolution Network, etc.; in this respect, the embodiments of this disclosure do not limit it.

[0068] S102. The object-oriented prediction of the point cloud to be segmented is performed through the second backbone network to obtain a multi-level point cloud including multiple levels of object-oriented labels; the multiple levels of object-oriented labels are used to characterize the shape of each instance, as well as internal and external information.

[0069] In this embodiment of the disclosure, the instance segmentation device can input the point cloud to be segmented into a second backbone network. The second backbone network can perform objectification prediction at multiple levels on the point cloud to be segmented to obtain a multi-level point cloud. Each point in the multi-level point cloud includes an objectification label of the corresponding level. The objectification labels at multiple levels can characterize the shape of each instance predicted by the second backbone network for the point cloud to be segmented, as well as other information that distinguishes the internal and external aspects of each instance, namely, the internal information of each instance and the external information of each instance.

[0070] It should be noted that the second backbone network is trained using a multi-level training point cloud with multiple levels of object labels. The number of levels of the object labels in the multi-level point cloud is the same as the number of levels of the object labels in the multi-level training point cloud, and the labels at each level are also the same. Thus, after obtaining the multi-level point cloud, the instance segmentation device can distinguish the core points within the core region and the edge points within the edge region of each instance. In this way, each instance is distinguished from other instances, and the boundary of each instance is perceived.

[0071] S103. Based on the centroid offset vector, move each point in the multi-level point cloud to its corresponding prediction center to obtain the influence point cloud;

[0072] In this embodiment of the disclosure, after obtaining the centroid offset vector, the instance segmentation device can use the center point regression method to move each point in the multi-level point cloud to its corresponding prediction center to obtain the influence point cloud; that is, the influence point cloud is a multi-level point cloud after each point has been moved, and the influence point cloud includes multiple levels of object labels.

[0073] In this embodiment of the disclosure, the instance segmentation device can use each point in the multi-level point cloud plus the corresponding centroid offset vector to obtain each influence point in the influence point cloud; in this way, the instance segmentation device can move the points in the multi-level point cloud toward the prediction center to obtain the influence point cloud corresponding to the multi-level point cloud, referring to formula (1).

[0074] x m '=x m +d m Formula (1)

[0075] The multi-level point cloud contains M points; m is a positive integer, and m∈M; x m For any point in a multi-level point cloud, d m For x m The corresponding centroid offset vector, x m 'for x m The corresponding points of influence.

[0076] For example, such as Figure 2 As shown, moving point clouds A, B, C, D, and E towards their respective prediction centers yields point clouds A', B', C', D', and E'. The figure shows that, through this movement, points in the edge regions of point clouds A'-E' become more marginalized compared to those in AE; points in the core regions of point clouds A'-E' cluster more towards the core regions compared to those in AC.

[0077] S104. Based on the semantic prediction results, cluster the core points in the influence point cloud to obtain at least one core point cloud for each semantic category; the core points are points with object labels of a preset core level.

[0078] In this embodiment of the disclosure, the instance segmentation device can cluster the core points in the influence point cloud according to the semantic category of each point in the semantic prediction result, to obtain at least one core point cloud for each semantic category; here, at least one core point cloud for each semantic category corresponds to at least one instance for each semantic category; the core points in the influence point cloud are labels with preset core levels among multiple levels of object-oriented labels. The multiple levels include preset core levels and preset edge levels.

[0079] For example, the multiple levels of object labels include 5, namely 0, 1, 2, 3 and 4. Among them, the preset core level object labels are 3 and 4, and 0-2 are the preset edge level object labels.

[0080] It should be noted that the instance segmentation device can use a clustering algorithm to cluster the core points; here, the clustering algorithm can be a breadth-first search algorithm (BFS), a PageRank algorithm, a graph algorithm such as the proximity centrality algorithm, or a density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN); the clustering algorithm can be set as needed, and this embodiment of the disclosure does not limit it.

[0081] S105. Cluster the edge points in the influence point cloud into the target core point cloud of the corresponding semantic category to obtain the instance segmentation result; the core points are pseudo-instances with preset edge level labels.

[0082] In this embodiment of the disclosure, after obtaining at least one core point cloud for each semantic category, the instance segmentation device can cluster each edge point into the target core point cloud in the corresponding category's at least one core point cloud, thereby completing instance segmentation and obtaining instance segmentation results; wherein, each edge point and the corresponding target core point cloud belong to the same semantic category in the semantic prediction results.

[0083] In this embodiment of the disclosure, the instance segmentation device can calculate the influence value between edge points and each core point cloud of the corresponding semantic category based on the distribution of the influence point cloud; thus, the instance segmentation device can calculate the influence value between each edge point in the influence point cloud and at least one core point cloud of the same semantic category, and cluster the edge points in the multi-level point cloud into the target core point cloud according to the magnitude of the influence value.

[0084] In this embodiment of the disclosure, the instance segmentation device can take the core point cloud with the largest influence value between itself and the corresponding edge point among at least one core point cloud of the same semantic category as the target core point cloud, and cluster each edge point into the target core point cloud, thereby completing the classification of the point cloud, i.e. instance segmentation.

[0085] In this embodiment of the disclosure, the influence value between an edge point and at least one core point cloud of the same semantic category is inversely correlated with the distance between the edge point and at least one core point cloud.

[0086] In some embodiments of this disclosure, the influence value between an edge point and a core point cloud of the same semantic category can be the influence value between the edge point and the center point of the core point cloud.

[0087] In some embodiments of this disclosure, the influence value between an edge point and a core point cloud of the same semantic category can be the average of the influence values ​​between the edge point and all points in the core point cloud.

[0088] Here, the influence value between edge points and core point clouds can be set as needed, and this disclosure does not impose any restrictions on this.

[0089] It should be noted that this method of distinguishing between core points and edge points, and using different strategies to cluster core points and edge points to complete instance segmentation, can be called an asymmetric object reasoning algorithm.

[0090] For example, Figure 3 An influence map is shown, such as Figure 3 As shown, the influence map includes points with three levels of objectivity labels: high objectivity points, medium objectivity points, and low objectivity points. High objectivity points are located at the core of the instance, medium objectivity points are located at the secondary core of the instance, and low objectivity points are located at the edge of the instance. High objectivity points and medium objectivity points can be clustered first as core points, and then low objectivity points can be clustered last.

[0091] It is understandable that the instance segmentation device includes a first backbone network and a second backbone network. Since both the first and second backbone networks are trained based on a first training point cloud, and the first training point cloud is determined based on a semantic standard point cloud, the training of the first and second backbone networks in the instance segmentation device does not use pre-labeled instance tags. This reduces the workload of instance labeling, improves network training efficiency, and thus improves the efficiency of instance segmentation. Furthermore, since the instance segmentation device clusters core points located in the core region of the instance and edge points located in the edge region separately, it improves the accuracy of edge point clustering, thereby improving the accuracy of instance segmentation.

[0092] In some embodiments of this disclosure, step S105 involves clustering edge points in the influence point cloud into the target core point cloud corresponding to the semantic category to obtain the instance segmentation result, such as... Figure 4 As shown, it may include: S201-S203.

[0093] S201. Calculate the influence value between each edge point and at least one core point cloud of the corresponding semantic category to obtain at least one influence value for each edge point; the influence value is used to characterize the average influence value between an edge point and all points in a core point cloud of the corresponding semantic category.

[0094] In this embodiment of the disclosure, the instance segmentation device clusters the cores in the influence point cloud to obtain at least one core point cloud for each semantic category. Then, it can calculate at least one influence value between the edge points in the influence point cloud and at least one core point cloud of the same semantic category to obtain at least one influence value for each edge point.

[0095] In this embodiment of the disclosure, the instance segmentation device can calculate the influence value between each edge point and each core point in a corresponding core point cloud, obtain each influence value, and then calculate the average of each influence value to obtain the influence value between each edge point and a corresponding core point cloud, thereby obtaining at least one influence value between each edge point and at least one core point cloud of the corresponding semantic category.

[0096] In this embodiment of the disclosure, the influence value between any edge point j and the core point cloud A can be calculated by formula (2).

[0097]

[0098] Among them, the core point cloud A includes P A Let i be any point in the core point cloud A, where i ∈ P. A w ij The influence value between edge point j and the i-th point in the core point cloud A can be calculated using formula (3). That is, when there are K core point clouds (A, B, ..., K) belonging to the same semantic category as edge point j, the instance segmentation device can obtain K influence values ​​for edge point j, expressed as...

[0099]

[0100] Among them, P coreLet A represent the core point cloud, and ε be a hyperparameter. ε can be set as needed; the larger ε is, the larger the influence range of a point, and the more points it influences. It can be seen that for any two points in the core point cloud, the influence value between the two points is not considered, i.e., the influence value between the two points is 0. The greater the distance between edge point j and the i-th point in the core point cloud A, the smaller their influence value.

[0101] In this embodiment of the disclosure, in order to promote the absorption of unclustered edge points by the core point cloud, the instance segmentation device needs to calculate the influence value between each core point cloud and other points within the same semantic category.

[0102] For example, such as Figure 5 The core point influence diagram shown has four influence values ​​between core point cloud A and points e1-e4, represented by four arrows. The closer a point is to core point cloud A, the thicker the arrow, indicating greater influence. In other words, among points e1-e4, the influence between core point cloud A and point e1 is the greatest. Similarly, core point cloud B has five influence values ​​between core point cloud B and points e5-e9, represented by five arrows. The closer a point is to core point cloud B, the thicker the arrow, indicating greater influence. Again, among points e5-e9, the influence between core point cloud B and point e9 is the greatest.

[0103] S202. Take the core point cloud corresponding to the maximum value of at least one influence value of each edge point as the target core point cloud corresponding to each edge point.

[0104] In this embodiment of the disclosure, after obtaining at least one influence value for each edge point, the instance segmentation device can use the influence core point cloud corresponding to the maximum value among the at least one influence value of each edge point as the target core point cloud. For example, if the influence value of the influence core point cloud C is the largest among the K influence values ​​of edge point j, it means that the target core point cloud of edge point j is C.

[0105] S203. Cluster each edge point into the corresponding target core point cloud to obtain at least one instance segmentation point cloud for each semantic category, thereby obtaining the instance segmentation result.

[0106] In this embodiment of the disclosure, after determining the target core point cloud of each edge point, the instance segmentation device can cluster each edge point into the corresponding target core point cloud to obtain at least one instance segmentation point cloud for each semantic category, thereby obtaining the instance segmentation result.

[0107] For example, Figure 6A This diagram illustrates the effect of clustering in a related technique; for example... Figure 6AAs shown, it can be seen that the clustering method based solely on radius search in related technologies may lead to undersegmentation and oversegmentation. That is, different instances A and B are not separated, while points belonging to instance B are oversegmented and not clustered into instance B. Figure 6B This is a schematic diagram illustrating an optional clustering effect provided in an embodiment of the present disclosure, such as... Figure 6B As shown, during the clustering process, since the core points and edge points are distinguished, the core point clouds a and b can be clustered first, and then the edge points a1 next to a and b1 next to b can be clustered to finally obtain instance A and instance B.

[0108] In some embodiments of this disclosure, the asymmetric object reasoning algorithm can be embodied by the following algorithm:

[0109] Input: Scene point cloud X∈R N×3 Semantic label S∈R N×1 Centroid offset vector D∈R N×3 Multi-level object-oriented label O∈R N×1 .

[0110] Output: Pseudo-instance tag C

[0111]

[0112] As can be seen from steps 1-4, the instance segmentation device only processes points N in the scene point cloud that belong to the foreground category. fg Process the scene, ignoring background categories. Taking an indoor scene as an example, background categories include walls and floors. Foreground categories include tables, chairs, cabinets, etc. in the room. (Step 5) It is the highest level in the marginal level; for example, If the value is 1, then the level of the core point is 2-4; that is, steps 5-8 are to distinguish between edge points and core points; step 9 is to move the point cloud to the prediction center of the corresponding instance; step 10 is to cluster the core points into K core point clouds by instance; steps 11-16 are to calculate the influence map and determine the core point cloud with the greatest influence on the edge points based on the influence map; step 17 is to move the edge point P j Clustering to the core cloud with the greatest influence In step 18, the identifier ID of each instance point cloud is obtained, and step 19 returns the instance label, completing instance segmentation.

[0113] Understandably, the instance segmentation device can cluster edge points into the core point cloud with the highest influence value based on the influence value between each edge point in the influence point cloud and the entire core point cloud, thereby improving the accuracy of edge point clustering and thus improving the accuracy of instance segmentation.

[0114] In some embodiments of this disclosure, the semantic prediction and centroid offset prediction of the acquired point cloud to be segmented are performed on the first backbone network in S101 to obtain the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented, as shown in the previous implementation. Figure 7 As shown, it may include: S301-S304.

[0115] S301. Cluster the original semantic point cloud to obtain the initial pseudo-instance point cloud.

[0116] In this embodiment of the disclosure, after obtaining the original semantic point cloud, the instance segmentation device can cluster the original semantic point cloud to obtain an initial pseudo instance point cloud; wherein, the original semantic point cloud is a point cloud with semantic labels.

[0117] In this embodiment of the disclosure, the instance segmentation device can cluster the original semantic point cloud according to the semantic tags using a clustering algorithm to obtain an initial pseudo instance point cloud; the initial pseudo instance point cloud contains the pseudo instance tags obtained by clustering.

[0118] In some embodiments of this disclosure, the clustering algorithm can be the BFS algorithm; that is, the instance segmentation device can use any point in each semantic category of the original semantic point cloud as a seed point, with the seed point as the center and r as the search radius, and search point by point in the same semantic category in the original semantic point cloud, and cluster the points within the radius r; then the points clustered with the seed point are used as new seed points, and the search continues with a radius of r until all points in the original semantic point cloud have been searched, thus completing the initial instance segmentation of the original semantic point cloud and obtaining the initial pseudo instance point cloud.

[0119] S302. Based on the number of points contained in each pseudo-instance in the initial pseudo-instance point cloud, filter the initial pseudo-instance point cloud to obtain the first training point cloud.

[0120] In this embodiment of the disclosure, after obtaining the initial pseudo-instance point cloud, the instance segmentation device can filter the initial pseudo-instance point cloud by the number of points contained in each pseudo-instance, and obtain the first training point cloud.

[0121] In this embodiment, a pseudo-instance with an excessive number of points may be an under-segmented instance, meaning multiple instances are clustered together to form a single pseudo-instance; a pseudo-instance with an insufficient number of points may be an over-segmented instance, meaning points that belong to only a portion of an instance. In other words, both excessive and insufficient points in the initial pseudo-instance point cloud can result in low-confidence pseudo-instances. Therefore, the instance segmentation device needs to filter out pseudo-instance point clouds with excessive or insufficient points to obtain a first training point cloud with high confidence. Here, the first training point cloud may include multiple high-confidence instance point clouds. For example, the first training point cloud may include point clouds of chair 1, chair 2, table, bed, lamp 1, and lamp 2.

[0122] Understandably, by filtering out pseudo-instance point clouds with low confidence from the initial pseudo-instance point cloud and obtaining the first training point cloud with high confidence, the accuracy of instance labels in the first training point cloud is improved, thereby improving the accuracy of the first backbone network and the second backbone network.

[0123] In some embodiments of this disclosure, in step S302, the initial pseudo-instance point cloud is filtered according to the number of points contained in each pseudo-instance in the initial pseudo-instance point cloud to obtain the first training point cloud, as follows: Figure 5 As shown, it may include: S401-S402.

[0124] S401. Sort the number of points in the point cloud of each pseudo-instance for each semantic category to obtain the order of the number of points for each semantic category.

[0125] In this embodiment of the disclosure, after obtaining the initial pseudo-instance point cloud, the instance segmentation device can sort the number of points in each pseudo-instance point cloud of each semantic category according to the semantic labels in the original semantic point cloud, and obtain the order of the number of points in each semantic category.

[0126] In some embodiments of this disclosure, the instance segmentation device can sort the number of points in each pseudo-instance point cloud of each semantic category in the initial pseudo-instance point cloud in ascending order to obtain the ascending order of the number of points in each semantic category; or it can sort the number of points in each pseudo-instance point cloud of each semantic category in descending order to obtain the descending order of the number of points in each semantic category. The sorting method can be set as needed, and this disclosure does not limit it.

[0127] S402. From the quantity order of each semantic category, select pseudo-instance point clouds within a preset quantity order range to obtain the training point cloud of each semantic category. The first training point cloud includes the training point cloud of each semantic category.

[0128] In this embodiment of the disclosure, after obtaining the quantity order of each semantic category, the instance segmentation device can select pseudo-instance point clouds within a preset quantity order range from the quantity order of each semantic category to obtain the training point cloud of each semantic category, thereby obtaining the first training point cloud.

[0129] In some embodiments of this disclosure, the preset quantity order range can be an ordinal range in the quantity order.

[0130] For example, the preset quantity order range can be the percentage range of ordinal numbers occupied in the quantity order, centered on the quantity center value. For instance, the initial pseudo-instance point cloud includes 5 pseudo-instance point clouds belonging to semantic category 1, where pseudo-instance point cloud 1 has 17289 points, pseudo-instance point cloud 2 has 3526 points, pseudo-instance point cloud 3 has 5564 points, pseudo-instance point cloud 4 has 4891 points, pseudo-instance point cloud 5 has 15 points, and pseudo-instance point cloud 6 has 32 points. The instance segmentation device sorts the points of the pseudo-instance point clouds of semantic category 1 in ascending order, resulting in the quantity order of 15, 32, 3526, 4891, 5564, and 17289. The preset quantity order range is 33%, meaning that the first training point cloud can include pseudo-instance point cloud 2 and pseudo-instance point cloud 3.

[0131] Understandably, by using the quantity order and the preset quantity order range, pseudo-instance point clouds with a number of points close to the center value can be selected from the initial pseudo-instance point cloud, thereby filtering out point clouds with too many or too few points. In this way, the initial pseudo-instance point cloud can be filtered quickly to obtain the first training point cloud.

[0132] S303. The initial second backbone network is trained based on the first training point cloud to obtain the second backbone network.

[0133] In this embodiment of the disclosure, after obtaining the first training point cloud, the instance segmentation device can train the initial second backbone network based on the first training point cloud, and after the training is completed, the second backbone network can be obtained.

[0134] In some embodiments of this disclosure, the instance segmentation device can reassemble the pseudo-instance point cloud in the first training point cloud to obtain a reassembled training point cloud; the reassembled training point cloud can represent a scene randomly generated by multiple sets of high-confidence pseudo-instance point clouds; and the initial second backbone network is trained based on the reassembled training point cloud to obtain the second backbone network.

[0135] In some embodiments of this disclosure, in step S203, the initial second backbone network is trained based on the first training point cloud to obtain the implementation of the second backbone network, such as... Figure 9 As shown, it may include: S501-S504.

[0136] S501. Compress the first training point cloud to obtain the first compressed point cloud.

[0137] In this embodiment of the disclosure, the instance segmentation device can reduce the coordinates of each point in the first training point cloud by the same ratio to achieve point cloud compression and obtain a first compressed point cloud. Here, the same ratio is preset, for example, 3:1, 4:1, and 2:1, etc.; the ratio value of the same ratio can be set as needed, and this embodiment of the disclosure does not limit it.

[0138] In some embodiments of this disclosure, the ratio is 2:1. The instance segmentation device can divide the coordinates of each point in the first training point cloud by 2 to obtain the coordinates of each point in the first compressed point cloud, thereby obtaining the first compressed point cloud. The first compressed point cloud includes multiple compressed pseudo-instance point clouds, and the multiple compressed pseudo-instance point clouds in the first compressed point cloud correspond one-to-one with the multiple pseudo-instance point clouds in the first training point cloud. Compared with the corresponding pseudo-instance point cloud in the first training point cloud, the shape of each compressed pseudo-instance point cloud in the first compressed point cloud remains unchanged, but its volume is reduced.

[0139] S502. Based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud, determine the multi-level training point cloud corresponding to each pseudo-instance point cloud; the multi-level training point cloud includes multiple levels of object labels.

[0140] In this embodiment of the disclosure, after obtaining the first compressed point cloud, the instance segmentation device can calculate the distance between each point in each pseudo-instance point cloud and the corresponding compressed pseudo-instance point cloud, thus obtaining the compressed distance between each point in each pseudo-instance point cloud and the corresponding compressed pseudo-instance point cloud. The number of compressed distances in each pseudo-instance point cloud is the same as the number of points in each pseudo-instance point cloud.

[0141] Understandably, since the compression distance between each point in each pseudo-instance point cloud and the compressed pseudo-instance point cloud contains the shape information of each pseudo-instance point cloud, the second backbone network can better perceive the shape of pseudo-instances when multi-level training point clouds are used for training the second backbone network, thus improving the accuracy of the second backbone network in predicting object labels.

[0142] In some embodiments of this disclosure, the instance segmentation device can calculate the distance between each point in each pseudo-instance point cloud and all points in the corresponding compressed pseudo-instance point cloud to obtain multiple distances for each point; and then take the average of the multiple distances as the compressed distance between each point in each pseudo-instance point cloud and the corresponding compressed pseudo-instance point cloud.

[0143] In some embodiments of this disclosure, in step S502, the implementation of the multi-level training point cloud corresponding to each pseudo-instance point cloud is determined based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud. Figure 10 As shown, it may include: S601-S603.

[0144] S601. Calculate the distance between each point in each pseudo-instance point cloud and each point in the corresponding compressed pseudo-instance point cloud, and obtain multiple distances corresponding to each point in each pseudo-instance point cloud.

[0145] In this embodiment of the disclosure, the instance segmentation device can calculate the distance between each point in each pseudo-instance point cloud and all points in the corresponding compressed pseudo-instance point cloud, thereby obtaining multiple distances corresponding to each point in each pseudo-instance point cloud; wherein, the number of multiple distances corresponding to each point in each pseudo-instance point cloud is the same as the number of points contained in each pseudo-instance point cloud.

[0146] S602. Take the minimum distance among the multiple distances corresponding to each point in each pseudo-instance point cloud as the compression distance between each point in each pseudo-instance point cloud and the corresponding compressed point cloud, and obtain the multiple compression distances of each pseudo-instance point cloud.

[0147] In this embodiment of the disclosure, after obtaining multiple distances corresponding to each point in each pseudo-instance point cloud, the instance segmentation device can determine the minimum distance from the multiple distances corresponding to each point in each pseudo-instance point cloud as the compression distance between each point in each pseudo-instance point cloud and the corresponding compressed point cloud, thereby obtaining multiple compression distances corresponding to multiple points in each pseudo-instance point cloud.

[0148] S603. Based on the multiple compression distances of each pseudo-instance point cloud, determine the multiple levels of object labels for each pseudo-instance point cloud, and obtain the multi-level training point cloud corresponding to each pseudo-instance point cloud.

[0149] In this embodiment of the disclosure, the instance segmentation device can determine multiple levels of object labels for each pseudo-instance point cloud based on multiple compression distances of each pseudo-instance point cloud, thereby obtaining a multi-level training point cloud corresponding to each pseudo-instance point cloud.

[0150] In this embodiment of the disclosure, the instance segmentation device can sort multiple compressed distances of each instance point cloud to obtain a distance order; classify the distance order according to multiple preset distance order ranges corresponding to multiple levels to obtain multiple levels of object labels; wherein the sum of multiple preset distance order ranges is 1.

[0151] For example, a pseudo-instance point cloud P in the first training point cloud. X It contains S points, and the point cloud P XThe coordinates X contain the coordinates of S points, denoted as X = {x1, x2, ..., x...} S}∈R S×3 The instance segmentation device can segment P X Compression is performed to obtain point cloud P. X Compressed point cloud P x Compressed point cloud P x The x-coordinate also includes the coordinates of S points, denoted as Then, from P X Each point in the compressed point cloud P x Find the point cloud P from the distance of each point in the cloud. X The nearest distance to each point in the point cloud is used as P. X S points in the compressed point cloud P x The compression distance is calculated, resulting in a compression distance list T. The compression distance list T includes the distances from S points in the point cloud to the compressed point cloud P. x The distance is denoted as T = {T1, T2, ..., T}. S}∈R S×1 Multiple levels of object-oriented label values. ob -ID and the corresponding multiple preset distance order ranges are shown in formula (4); where, the object label value label ob -IDs of 2-4 can be preset core levels, and the object-oriented label value can be used. ob -IDs of 0-1 can be preset edge levels.

[0152]

[0153] In this embodiment of the disclosure, the number of object tags of multiple levels can be set as needed, and the multiple preset distance order ranges corresponding to the object tags of multiple levels can also be set as needed; wherein, the size of the multiple preset distance order ranges can be the same or different; this embodiment of the disclosure does not limit the number, identifier and corresponding preset distance order range of object tags of multiple levels.

[0154] Understandably, by sorting multiple compressed distances and multiple preset distance order ranges, multiple levels of object labels can be quickly determined, thereby quickly obtaining multi-level training point clouds corresponding to each pseudo-instance point cloud.

[0155] S503. Based on the multi-level training point cloud corresponding to each pseudo-instance point cloud, perform point cloud recombination to obtain the recombined training point cloud.

[0156] In this embodiment of the disclosure, after obtaining the multi-level training point cloud corresponding to each pseudo-instance, the instance segmentation device can perform point cloud recombination based on the multi-level training point cloud corresponding to each pseudo-instance to obtain the recombined training point cloud.

[0157] In some embodiments of this disclosure, the instance segmentation device can select at least one multi-level training point cloud from all the multi-level training point clouds corresponding to all pseudo instances, and perform spatial rotation, spatial position transformation and other processing on the at least one multi-level training point cloud to obtain a recombined training point cloud.

[0158] For example, Figure 11A The four multi-level training point clouds corresponding to the four pseudo-instance point clouds are shown; Figure 11B The diagram illustrates a reconstructed training point cloud obtained by recombining five multi-level training point clouds corresponding to five pseudo-instance point clouds. Figure 11B The virtual scene corresponding to the recombined training point cloud, such as Figure 11C As shown.

[0159] In some embodiments of this disclosure, the implementation of point cloud reorganization based on the multi-level training point cloud corresponding to each pseudo-instance point cloud in step S503 to obtain the reorganized training point cloud may include:

[0160] S5031. According to the preset point cloud attributes in the recombination template, the multi-level training point cloud is recombined to obtain the recombined training point cloud; the preset point cloud attributes include: preset point cloud quantity, preset point cloud volume and preset point cloud position; there are no intersecting regions between the preset point cloud positions.

[0161] In this embodiment of the disclosure, the preset point cloud attributes in the reconstructed template may include: preset point cloud quantity, preset point cloud position, and preset point cloud volume corresponding to the preset point cloud position; wherein, the preset point cloud quantity is used to represent the number of multi-level training point clouds included in the reconstructed training point cloud, the preset point cloud position is used to represent the position corresponding to each point cloud in the preset point cloud quantity, and the preset point cloud volume is used to represent the volume of the multi-level training point cloud at each preset point cloud position; in some embodiments, when the point cloud density is uniform to a certain value, the point cloud volume can be characterized by the number of points in the point cloud.

[0162] It should be noted that the preset point cloud positions in the recombined template do not intersect in space.

[0163] For example, Figure 12 An optional recombination template is shown, comprising nine preset point clouds corresponding to nine preset point cloud positions: 0-8. Position 0 is the center position; the preset point cloud volume at each position is different. The instance segmentation device can obtain nine multi-level training point clouds from the first training point cloud according to the nine different point cloud volumes. These nine multi-level training point clouds are then placed at their corresponding nine preset point cloud positions to obtain the recombined training point clouds.

[0164] Understandably, the instance segmentation device can perform point cloud reconstruction based on the reconstruction template, thereby improving the efficiency of point cloud reconstruction.

[0165] S504. The initial second backbone network is trained using the recombined training point cloud to obtain the second backbone network and the multi-level predicted point cloud output by the second backbone network.

[0166] In this embodiment of the disclosure, after obtaining the reconstructed training point cloud, the instance segmentation device can use the object labels of multiple levels in the reconstructed training point cloud as supervision data, and use the reconstructed training point cloud with the object labels of multiple levels removed as input data to train the initial second backbone network to obtain the second backbone network.

[0167] It is understandable that since a large number of virtual point cloud scenes can be randomly generated through point cloud recombination, the recombination training point cloud can contain a large amount of training data; thus, the accuracy of the second backbone network can be improved.

[0168] In some embodiments of this disclosure, the implementation of training the initial second backbone network using the recombined training point cloud in S504 to obtain the second backbone network may include: S701-S705.

[0169] S701. Obtain the second backbone network updated in the (k-1)th iteration; where k is a positive integer; when k equals 1, the second backbone network updated in the (k-1)th iteration is the initial second backbone network.

[0170] In this embodiment of the disclosure, the instance segmentation device updates the initial second backbone network multiple times. The second backbone network updated in the (k-1)th time indicates that the initial second backbone network has been updated k-1 times. When k is 1, the second backbone network updated in the (k-1)th time indicates that the initial second backbone network has been updated 0 times, that is, no update. In other words, when k is 1, the second backbone network updated in the (k-1)th time is the initial second backbone network.

[0171] S702. The second backbone network updated in the (k-1)th iteration is used to extract features from each point of the recombined training point cloud, and the second point cloud features of each point in the kth iteration are obtained.

[0172] In this embodiment of the disclosure, after obtaining the second backbone network updated for the (k-1)th time, the instance segmentation device can extract features from each point of the reconstructed training point cloud through the second backbone network updated for the (k-1)th time, and obtain the second point cloud features of each point for the kth time.

[0173] S703. Perform a second linear transformation on the k-th second point cloud feature of each point to obtain the k-th second linear transformation feature; the dimension of the k-th second linear transformation feature corresponds one-to-one with the number of levels of the object label at multiple levels.

[0174] In this embodiment of the disclosure, after obtaining the second point cloud feature of each point at the kth time, the instance segmentation device can perform a second linear transformation on the second point cloud feature of each point at the kth time to obtain a second linear transformation feature; wherein, the dimension of the second linear transformation feature at the kth time corresponds one-to-one with the number of levels of the object label of the multiple levels; the value of each dimension of the second linear transformation feature at the kth time represents the score of the corresponding level label.

[0175] S704. Calculate the kth second cross-entropy between the kth second linear transformation feature of each point and the corresponding point's true objectivity label, and obtain the mean of the kth second cross-entropy of all points as the kth objectivity loss.

[0176] In this embodiment of the disclosure, the instance segmentation device can select the maximum value from all dimensions of the second linear change feature of the kth time for each point as the second target dimension value of the corresponding point, calculate the kth cross-entropy between the second target dimension values ​​of all points and the corresponding level's instance label value of the kth time, and use the mean of the kth cross-entropy of all points as the objectivity loss L. objectness Refer to formula (5)

[0177]

[0178] The reconstructed training point cloud consists of N1 points. For any point n, the feature of the k-th second linear transformation is q. n The real object-oriented label is o n .

[0179] For example, for point n, the k-th second linear change feature q n The value is [0.3, 0.1, 0.1, 0.1, 0.9]; where the label of point n is... ob - The predicted score for ID 0 is 0.3, and the label for point n is... ob - The predicted score for IDs 1-3 is 0.1, and the label for point n is... ob - The predicted score for ID 4 is 0.9; the true object label for point n is o. n [0,0,0,0,1] represents the actual label of point n. ob -ID is 4.

[0180] S705. If the difference between the object loss of the kth time and the object loss of the (k-1)th time is continuously less than or equal to the second difference threshold, and the number of consecutive differences is greater than or equal to the second preset number, training is stopped, and the second backbone network updated in the (k-1)th time is used as the second backbone network; and the multi-level point cloud predicted by the second backbone network updated in the (k-1)th time is used as the multi-level predicted point cloud.

[0181] In this embodiment of the disclosure, after obtaining the object-specific loss of the kth time, the instance segmentation device can subtract the object-specific loss of the (k-1)th time from the object-specific loss of the kth time to obtain the difference of the object-specific loss of the kth time. If the difference of the object-specific loss of the kth time is less than or equal to the second difference threshold for the Hth consecutive time, and H is greater than or equal to the second preset number of times, training is stopped. The second backbone network updated in the (k-1)th time is taken as the trained second backbone network. The second difference threshold and the second preset number of times can be set according to actual needs, and this embodiment of the disclosure does not limit them.

[0182] It is understandable that the difference in objectivity loss at the k-th update is determined by the second backbone network updated at the (k-1)-th update. In this way, the instance segmentation device can obtain the difference in objectivity loss between the second backbone network updated each time and the second backbone network updated in the previous update. Based on the condition that the difference in objectivity loss is stable for multiple consecutive times, training is stopped, so that the second backbone network can reach the highest accuracy and improve the accuracy of the second backbone network in predicting multi-level point clouds.

[0183] In some embodiments of this disclosure, the calculation of the k-th second cross-entropy between the k-th second linear transformation feature of each point and the corresponding point's true objectivity label in S704, and obtaining the mean of the k-th second cross-entropy of all points as the implementation after the k-th objectivity loss, may include:

[0184] S706. If the difference in the object-oriented loss at the kth time is greater than the preset difference threshold, or if the difference in the object-oriented loss at the kth time is continuously less than or equal to the second preset difference threshold, and the number of consecutive occurrences is less than the second preset number of occurrences, the second backbone network at the (k-1)th time is updated according to the object-oriented loss at the kth time, and the second backbone network updated at the kth time is obtained.

[0185] In this embodiment of the disclosure, the instance segmentation device can update the second backbone network of the (k-1)th iteration based on the object-related loss of the kth iteration when the difference in object-related loss of the kth iteration is greater than a preset difference threshold, thereby obtaining the second backbone network of the kth iteration.

[0186] In this embodiment of the disclosure, the instance segmentation device may update the second backbone network of the (k-1)th iteration based on the object loss of the kth iteration, when the difference of the object loss of the kth iteration is continuously less than or equal to the second preset difference threshold and the number of consecutive iterations is less than the second preset number of iterations, so as to obtain the second backbone network of the kth iteration.

[0187] It is understandable that the instance segmentation device can update the second backbone network for the (k-1)th time if the difference in the objectivity loss of the kth time does not meet the condition in S705, until the loss difference threshold of the second backbone network updated for the k+hth time is continuously less than or equal to the second preset difference threshold, and the number of consecutive times is greater than or equal to the second preset number, then training stops, and the second backbone network updated for the k+hth time is taken as the second backbone network; h is a positive integer.

[0188] In some embodiments of this disclosure, the implementation after performing the k-th update on the (k-1)-th second backbone network according to the k-th objectivity loss in step S706 to obtain the k-th updated second backbone network may include:

[0189] S707. If k is greater than or equal to the second update threshold, stop training and use the second backbone network updated in the kth update as the second backbone network.

[0190] In this embodiment of the disclosure, the second update number threshold is the maximum number of updates for the second backbone network. After obtaining the second backbone network updated for the kth time, the instance segmentation device can determine whether k is greater than or equal to the second update number threshold. If so, training is stopped; otherwise, the loss for the (k+1)th time is obtained. Referring to the steps in S705, the difference between the (k+1)th and the losses is used to determine whether the second backbone network should be trained again.

[0191] The second update count threshold can be set as needed, and this embodiment does not impose any restrictions on it.

[0192] Understandably, the second update threshold can limit the number of times the second backbone network can be trained, thereby improving the training efficiency of the second backbone network.

[0193] S304. The initial first backbone network is trained based on the first training point cloud and the original semantic point cloud to obtain the first backbone network.

[0194] In this embodiment of the disclosure, after obtaining the first training point cloud, the instance segmentation device can train the initial first backbone network based on the first training point cloud and the original semantic point cloud, and obtain the first backbone network after the training is completed.

[0195] In some embodiments of this disclosure, the instance segmentation device can use pseudo-instance labels in the first training point cloud and semantic labels in the original semantic point cloud as supervision data to train the initial first backbone network and obtain the first backbone network.

[0196] In some embodiments of this disclosure, the instance segmentation device can use pseudo-instance labels in the first training point cloud and semantic labels in the original semantic point cloud as supervision data to perform a first round of training on the initial first backbone network. Then, based on the semantic segmentation result and centroid offset vector output by the first backbone network after the first round of training, and the multi-level prediction point cloud output by the trained second backbone network, instance segmentation is performed to obtain updated pseudo-instance labels. Finally, using the updated pseudo-instance labels and semantic labels in the original semantic point cloud as supervision data, a second round of training is performed on the first backbone network after the first round of training.

[0197] It is understandable that the instance segmentation device can obtain the first training point cloud from the original semantic point cloud, and train the first backbone network and the second backbone network based on the first training point cloud, without using point clouds with instance labels for supervision. In this way, the manual cost of training data can be reduced and the efficiency of instance segmentation can be improved.

[0198] In some embodiments of this disclosure, in step S304, an initial first backbone network is trained based on the first training point cloud and the original semantic point cloud to obtain the implementation of the first backbone network, such as... Figure 13 As shown, it may include: S801-S803.

[0199] S801. Based on the pseudo-instance labels in the first training point cloud and the corresponding semantic labels in the original semantic point cloud, the initial first backbone network is trained in the first round to obtain the initial third backbone network, the third centroid offset vector, and the third semantic prediction result.

[0200] In this embodiment of the disclosure, after obtaining the first training point cloud, the instance segmentation device can use the pseudo-instance labels in the first training point cloud and the semantic labels in the original semantic point cloud as supervision data, and use the original training point cloud as training data to train the initial first backbone network, and obtain the trained network as the initial third backbone network; the centroid offset vector predicted by the initial third backbone network is the third centroid offset vector; and, based on the semantic prediction result predicted by the initial third backbone network, a third semantic prediction result is determined; the third semantic prediction result includes the predicted semantic labels.

[0201] In some embodiments of this disclosure, the implementation of S801, which involves performing a first round of training on the initial first backbone network based on the pseudo-instance labels in the first training point cloud and the corresponding semantic labels in the original semantic point cloud to obtain the initial third backbone network, the third centroid offset vector, and the third semantic prediction result, may include:

[0202] S901. Obtain the first backbone network updated in the (α-1)th time; where α is a positive integer; when α equals 1, the first backbone network updated in the (α-1)th time is the initial first backbone network.

[0203] In this embodiment of the disclosure, the instance segmentation device updates the initial first backbone network multiple times. The first backbone network updated in the (α-1)th time indicates that the initial first backbone network has been updated α-1 times to obtain the network. When α is 1, the first backbone network updated in the (α-1)th time indicates that the initial first backbone network has been updated 0 times, that is, no update. In other words, when α is 1, the second backbone network updated in the (α-1)th time is the initial first backbone network.

[0204] S902. Based on the first backbone network updated in the (α-1)th time, the semantic labels in the original semantic point cloud, and the pseudo-instance labels in the first training point cloud, the first semantic prediction result, the first semantic loss, and the first prediction offset vector are obtained in the (α-1)th time. The first offset loss is used to characterize the difference between the centroid of the instance predicted by the first backbone network updated in the (α-1)th time and the centroid of the real pseudo-instance. The first training point cloud contains real pseudo-instances.

[0205] In this embodiment of the disclosure, the instance segmentation device can perform semantic prediction on the original training point cloud using the first backbone network updated in the (α-1)th time and the semantic labels in the original semantic point cloud, to obtain the first semantic prediction result and the first semantic loss in the αth time.

[0206] In this embodiment of the disclosure, the instance segmentation device can perform centroid offset prediction on the original training point cloud using the first backbone network updated for the (α-1th time) and the pseudo-instance labels in the first training point cloud to obtain the first predicted offset vector.

[0207] In some embodiments of this disclosure, the implementation of obtaining the first semantic prediction result, the first semantic loss, and the first prediction offset vector based on the first backbone network updated for the (α-1)th time, the semantic labels in the original semantic point cloud, and the pseudo-instance labels in the first training point cloud in S902 may include: S1001-S1002.

[0208] S1001. The first backbone network updated for the α-1th time performs the α-th feature extraction on each original point in the original training point cloud to obtain the α-th first point cloud feature of each original point.

[0209] In this embodiment of the disclosure, the instance segmentation device can extract features from each point in the original training point cloud through the first backbone network updated for the (α-1)th time, to obtain the first point cloud feature F∈R for each original point at the αth time. N×dim .

[0210] Here, the dimension dim of the first point cloud feature can be set as needed. For example, the dimension of the first point cloud feature can be 128 dimensions, 64 dimensions, etc. This disclosure does not limit this.

[0211] S1002. Semantic prediction is performed on the first point cloud features of each original point through the first backbone network updated for the (α-1)th time, to obtain the first semantic prediction result for the αth time; based on the first point cloud features of each original point and the semantic label of the corresponding original point, the first semantic loss for the αth time is determined; and feature transformation is performed on the first point cloud features of each original point to obtain the first predicted centroid offset vector for the αth time of each original point.

[0212] In this embodiment of the disclosure, after obtaining the first point cloud features for the αth time, the instance segmentation device can perform semantic prediction on the first point cloud features for each original point through the first backbone network updated for the α-1th time, and obtain the first semantic prediction result for the αth time; the first semantic prediction result for the αth time includes the semantic label predicted by the first backbone network updated for the α-1th time.

[0213] In this embodiment of the disclosure, after obtaining the first point cloud feature of the αth time, the instance segmentation device can perform a first linear transformation on the first point cloud feature of the αth time for each original point to obtain the first linear transformation feature of the αth time; the dimension of the first linear transformation feature is the same as the number of categories of semantic labels in the original semantic point cloud; each dimension value of the first linear transformation feature represents the score of the corresponding semantic label category.

[0214] In this embodiment of the disclosure, the instance segmentation device can calculate the mean of the cross-entropy between the αth first linear transformation feature and the corresponding semantic label of each point, as the αth first semantic loss. The calculation of the first semantic loss can refer to formula (6).

[0215]

[0216] The original semantic point cloud consists of N² points. For any point n, the feature of the α-th first linear transformation is y. n The corresponding semantic tag is c n .

[0217] In this embodiment of the disclosure, after obtaining the first point cloud feature of each original point, the instance segmentation device can perform feature transformation on the first point cloud feature of each original point to obtain the first predicted centroid offset vector of each original point.

[0218] Here, the instance segmentation device can use a multi-layer perceptron (MLP) to perform feature transformation on the first point cloud features in the αth iteration, to obtain the αth predicted centroid offset vector D∈R. N×3 .

[0219] In some embodiments, the MLP can be a two-layer MLP, which transforms the first point cloud feature of each original point into the predicted centroid offset vector of the corresponding original point through the two-layer MLP.

[0220] It is understandable that after extracting the first point cloud features of each original point, the instance segmentation device can obtain the first semantic prediction result, the first semantic loss, and the first prediction offset vector based on the first point cloud features of each original point.

[0221] In some embodiments of this disclosure, the instance segmentation device can perform semantic prediction on the first point cloud features of each original point through the first backbone network updated for the (α-1)th time, and the obtained semantic prediction result can be directly used as the first semantic prediction result for the αth time.

[0222] In some embodiments of this disclosure, in S1002, the semantic prediction of the first point cloud features of each original point is performed by the first backbone network updated for the (α-1)th time, thereby obtaining the first semantic prediction result for the αth time. Figure 14 As shown, it may include: S10021-S10022.

[0223] S10021. Semantic prediction is performed on the first point cloud features of each original point through the first backbone network updated for the (α-1)th time, to obtain the initial semantic prediction point cloud for the αth time.

[0224] In this embodiment of the disclosure, the instance segmentation device performs semantic prediction on the first point cloud features of each original point through the first backbone network updated for the (α-1)th time, to obtain the initial semantic prediction point cloud for the αth time; each point in the initial semantic prediction point cloud for the αth time has a corresponding semantic feature vector.

[0225] S10022. By using average pooling, the semantic feature vector in each supervoxel of the initial semantic prediction point cloud of the αth time is smoothed to obtain the fine-tuned initial semantic prediction point cloud of the αth time as the first semantic prediction result.

[0226] In this embodiment of the disclosure, the supervoxels in any initial semantic prediction point cloud correspond to the pre-calculated supervoxels in the original semantic point cloud; the instance segmentation device can perform supervoxel clustering on the original semantic point cloud using a supervoxel clustering algorithm to obtain multiple supervoxels; the multiple supervoxels in the original semantic point cloud correspond one-to-one with the multiple supervoxels in the initial semantic prediction point cloud.

[0227] It should be noted that the supervoxel clustering algorithm can be a fuzzy C-means clustering method, a flow-constrained clustering algorithm, or an unsupervised oversegmentation method, etc. The supervoxel clustering algorithm can be set as needed, and this embodiment does not impose any restrictions.

[0228] In this embodiment of the disclosure, the instance segmentation device can calculate the average value of each semantic feature vector in each of the multiple supervoxels to obtain the average semantic vector of each supervoxel; and use the average semantic vector of each supervoxel as the semantic feature vector of each point in the corresponding supervoxel to obtain the fine-tuned initial semantic prediction point cloud of the αth time, which is used as the first semantic prediction result of the αth time.

[0229] Understandably, fine-tuning the semantic prediction results based on supervoxels can reduce the interference of noise in the point cloud, thereby improving the effect of subsequent instance segmentation.

[0230] S903. Based on the first semantic prediction result of the αth time and the original semantic point cloud, perform point cloud clustering to obtain the first instance prediction result of the αth time.

[0231] In this embodiment of the disclosure, after obtaining the αth first semantic prediction result, the instance segmentation device can cluster the original semantic point cloud according to the semantic label of each point in the αth first semantic prediction result to obtain the αth first instance prediction result. Here, the point cloud clustering method can be referred to the description in S103, and will not be repeated here.

[0232] S904. Based on the prediction result of the first instance in the αth iteration, the first training point cloud, and the first prediction offset vector in the αth iteration, determine the first offset loss.

[0233] In this embodiment of the disclosure, after obtaining the first instance prediction result and the first prediction offset vector, the instance segmentation device can determine the first offset loss based on the first instance prediction result, the first training point cloud, and the first prediction offset vector.

[0234] In some embodiments of this disclosure, the method for determining the first offset loss in S904 based on the αth first instance prediction result, the first training point cloud, and the αth first prediction offset vector is as follows: Figure 15 As shown, it may include: S1101-S1103.

[0235] S1101. Determine the average coordinates of all points in each instance in the αth first instance prediction result as the centroid vector of the αth first instance for each point in the corresponding instance.

[0236] In this embodiment of the disclosure, after obtaining the αth first instance prediction result, the instance segmentation device can use the average coordinates of all points of each instance in the αth first instance prediction result as the αth first instance centroid vector of each point in the corresponding instance.

[0237] S1102. Using the first instance centroid vector of the first instance corresponding to each point in the first training point cloud, subtract the coordinates of the corresponding point in the first instance prediction result to obtain the first vector difference of the first instance for each point, which is used as the first true centroid offset vector of the first instance corresponding to each point in the first training point cloud.

[0238] In this embodiment of the disclosure, after obtaining the αth first instance prediction result, the instance segmentation device can determine the αth first instance centroid vector of each point corresponding to the first training point cloud. Thus, the instance segmentation device can use the αth first instance centroid vector of each point corresponding to the first training point cloud to subtract the coordinate p of the corresponding point. n This yields the α-th first vector difference value corresponding to each point in the first training point cloud. This serves as the α-th first true centroid offset vector for each point corresponding to the first training point cloud.

[0239] S1103. Based on the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point, determine the αth first offset loss.

[0240] In this embodiment of the disclosure, after obtaining the first true centroid offset vector of the αth time for each point corresponding to the first training point cloud, the instance segmentation device can determine the first offset loss for the first time based on the first true centroid offset vector of the αth time for each point corresponding to the first training point cloud and the first predicted centroid offset vector of the αth time.

[0241] Understandably, the instance segmentation device can determine the true centroid offset vector based on the first instance prediction result of the αth time, and determine the first offset loss together with the true centroid offset vector and the predicted centroid offset vector of each point corresponding to the first training point cloud, so that the first backbone network can learn to predict the centroid offset vector for the first training point cloud with high confidence, thereby improving the accuracy of the centroid offset vector.

[0242] In some embodiments of this disclosure, in S1103, the implementation of the first offset loss for the αth time is determined based on the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point, such as... Figure 16 As shown, it may include: S1201-S1206.

[0243] S1201. Calculate the norm of the difference between the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point, and obtain the αth first difference norm of each point corresponding to the first training point cloud.

[0244] In this embodiment of the disclosure, the instance segmentation device can calculate the α-th first predicted centroid offset vector d for each point corresponding to the first training point cloud. n and the first true centroid offset vector of the αth time The norm of the difference between the two points is used as the first difference norm of the αth time for each point.

[0245] S1202. Calculate the average of the first difference norm of the αth time for each point corresponding to the first training point cloud to obtain the first sub-offset loss of the αth time.

[0246] In this embodiment of the disclosure, the instance segmentation device can calculate the average value of the first difference norm for all points corresponding to the first training point cloud, and obtain the first sub-offset loss L for the αth time. offset1 Refer to formula (7).

[0247]

[0248] Where n represents any point in the original semantic point cloud, and M n ∈{0,1} is a binary mask, and if n belongs to a point in the first training point cloud, then M n The value is 0 otherwise; that is, the first sub-offset loss L for the αth time is... offset1 Only the points corresponding to the first training point cloud are calculated.

[0249] S1203. Calculate the ratio between the first predicted centroid offset vector of the αth time for each point corresponding to the first training point cloud and the norm of the first predicted centroid offset vector of the corresponding point, as the first predicted ratio of the αth time; and calculate the ratio between the first actual centroid offset vector of the αth time for each point corresponding to the first training point cloud and the first true centroid offset vector of the corresponding point, as the first actual ratio of the αth time.

[0250] S1204. Determine the product of the αth first prediction ratio and the αth first actual ratio of each point corresponding to the first training point cloud, and obtain the αth first direction product of each point corresponding to the first training point cloud.

[0251] S1205. Calculate the average of the α-th direction product of each point corresponding to the first training point cloud to obtain the α-th second sub-offset loss.

[0252] In this embodiment of the disclosure, the instance segmentation device can calculate, for each point corresponding to the first training point cloud, the ratio between the norm of the αth first predicted centroid offset vector and the norm of the αth first predicted centroid offset vector, as the αth first prediction ratio; calculate the ratio between the norm of the αth first true centroid offset vector and the norm of the αth first true centroid offset vector, as the αth first actual ratio; then multiply the αth first prediction ratio and the αth first actual ratio to obtain the αth first directional product; subsequently, the averaging of the αth first directional products of all points can be used to obtain the αth second sub-offset loss L. offset2 Refer to formula (8).

[0253]

[0254] Among them, the norm of the first predicted centroid offset vector in the αth iteration and the norm of the first true centroid offset vector in the αth iteration are both A2 norms. It can be seen that through M... n The settings, the second sub-offset loss L for the αth time offset2 Only the points corresponding to the first training point cloud are calculated.

[0255] It should be noted that the second sub-offset loss L offset2 Characterizes the additional orientation loss based on cosine similarity.

[0256] S1206. Subtract the second sub-offset loss from the first sub-offset loss of the αth sub-offset to obtain the first offset loss of the αth sub-offset.

[0257] In this embodiment of the disclosure, the instance segmentation device obtains the first sub-offset loss L for the αth time. offset1 and the second sub-offset loss L of the αth time offset2 Then, the α-th first sub-offset loss L can be used. offset1 Subtract the second sub-offset loss L offset2 The first offset loss for the αth time is obtained, referring to formula (9).

[0258] L offset =L offset1 -L offset2 Formula (9)

[0259] Here, the first sub-offset loss L in formula (9) is... offset1 Second sub-offset loss L offset2 After expansion, it can be represented by formula (10).

[0260]

[0261] S905. The sum of the first semantic loss and the first offset loss of the αth time is taken as the first loss of the αth time.

[0262] In this embodiment of the disclosure, the instance segmentation device calculates the first semantic loss L for the αth time. semantic and the first offset loss L of the αth time offset Then, the α-th first semantic loss L can be... semantic and the first offset loss L of the αth time offset The sum of these values ​​serves as the α-th first loss. The α-th first loss is the α-th first semantic loss L. semantic and the first offset loss L of the αth time offset The combined loss is denoted as L. joint Refer to formula (11).

[0263] L joint =L offset +L semantic Formula (11)

[0264] Understandably, the instance segmentation device updates the first backbone network through a joint loss of semantic loss and offset loss for the centroid offset vector, so that the final first backbone network can simultaneously guarantee the accuracy of semantic prediction results and centroid offset vector prediction, thereby improving the accuracy of subsequent instance segmentation.

[0265] S906. If the difference between the αth first loss and the α-1th first loss is continuously less than or equal to the first difference threshold, and the number of consecutive losses is greater than or equal to the first preset number, training is stopped, and the α-1th updated first backbone network is obtained as the initial third backbone network; and the αth first prediction offset vector is obtained as the third centroid offset vector; and the αth first semantic prediction result is obtained as the third semantic prediction result.

[0266] In this embodiment of the disclosure, after obtaining the first loss for the αth time, the instance segmentation device obtains the first loss for the (α-1)th time, calculates the difference between the first loss for the αth time and the first loss for the (α-1)th time, and uses it as the difference of the first loss for the αth time.

[0267] In this embodiment of the disclosure, the instance segmentation device can determine that the output region of the first backbone network is stable and stop training when the first loss difference in the αth time is continuously less than or equal to the first difference threshold and the number of consecutive occurrences is greater than or equal to the first preset number of occurrences. The first backbone network updated in the α-1th time is then used as the initial third backbone network. The first prediction offset vector in the αth time is used as the third centroid offset vector, and the first semantic prediction result in the αth time is used as the third semantic prediction result.

[0268] Understandably, when the value of the first loss tends to stabilize, the difference between the current first loss and the previous first loss is small, which can be less than or equal to the first difference threshold, and the number of consecutive differences is greater than or equal to the first preset number. In this way, the initial third backbone network can achieve the highest accuracy, thereby improving the accuracy of the initial third backbone network in predicting the centroid offset vector and semantic segmentation results.

[0269] In some embodiments of this disclosure, the first backbone network may be trained using point clouds in a three-dimensional coordinate system during the first round of training.

[0270] In some embodiments of this disclosure, the first backbone network may ignore depth coordinates during the first round of training. For example, if the three-dimensional coordinates of a point are (x, y, z), the z coordinate may be ignored, and a two-dimensional point cloud may be used for training, or the z coordinate may be uniformly set to a single value.

[0271] It is understandable that ignoring depth coordinates during training can reduce interference caused by the sampling process. For example, considering that depth coordinates may cause a chair to be segmented into two instances, i.e., oversegmentation, this situation can be improved by ignoring depth coordinates, thereby improving the performance of the first backbone network in the first round of training, and thus improving the instance segmentation performance.

[0272] S802. Based on the third semantic prediction result and the third centroid offset vector, perform instance segmentation on the multi-level prediction point cloud output by the second backbone network to obtain the third training point cloud.

[0273] In this embodiment of the disclosure, after the instance segmentation device completes the first round of training on the initial first backbone network and obtains the initial third backbone network, it can perform instance segmentation on the multi-level prediction point cloud output by the second backbone network based on the third semantic prediction result and the third centroid offset vector output by the initial third backbone network to obtain the third training point cloud.

[0274] In some embodiments of this disclosure, the implementation of performing instance segmentation on the multi-level predicted point cloud output by the second backbone network based on the third semantic prediction result and the third centroid offset vector in S802 to obtain the third training point cloud may include: S1301-S1303.

[0275] S1301. Based on the third centroid offset vector, move each point in the multi-level prediction point cloud to its corresponding prediction center to obtain the influence prediction point cloud.

[0276] S1301. Based on the third semantic prediction results, cluster the core points in the influence prediction point cloud to obtain at least one third core point cloud for each semantic category.

[0277] S1303. Cluster the edge points in the influence prediction point cloud into the third target core point cloud of at least one third core point cloud of the corresponding semantic category to obtain the third training point cloud.

[0278] In the embodiments disclosed herein, the implementation of S1301-S1303 can be referred to the description of S103-S105, and will not be repeated here.

[0279] Understandably, the instance segmentation device can cluster the core points of the core region and the edge points of the edge region in the multi-level predicted point cloud predicted by the second backbone network in different ways, thereby reducing the under-segmentation problem of edge points and improving the accuracy of instance segmentation in the third training point cloud.

[0280] S803. Based on the pseudo-instance labels of the third training point cloud and the semantic labels of the original semantic point cloud, the initial third backbone network is trained in the second round to obtain the first backbone network.

[0281] In this embodiment of the disclosure, the instance segmentation device performs a second round of training on the initial third backbone network based on the third training point cloud and the original semantic point cloud to obtain the first backbone network. This method can be referred to in S801, which describes the implementation of the initial first backbone network based on the first training point cloud and the original semantic point cloud to obtain the initial third backbone network. It will not be repeated here.

[0282] Understandably, since the third training point cloud is based on the multi-level prediction point cloud and is the result of clustering core points and edge points respectively, it contains more training data than the first training point cloud. At the same time, the accuracy of the pseudo-instance labels is higher. By using the third training point cloud as new supervision data to replace the first training point cloud, the prediction accuracy of the first backbone network can be further improved.

[0283] This disclosure provides a schematic diagram of the training process of an instance segmentation network, as shown in the embodiments below. Figure 17As shown, the instance segmentation network includes 3D U-net backbone network 1 and 3D U-net backbone network 2. Solid arrows represent data input and output, dashed arrows represent the input of supervised data, and straight arrows represent the input of updated supervised data. The figure shows that by segmenting the original semantic point cloud using the BFS algorithm, an initial pseudo-instance point cloud can be obtained. Point cloud filtering based on the number of points in each pseudo-instance point cloud yields the first training point cloud. The first training point cloud and the original semantic point cloud serve as supervised data for the first round of training of 3D U-net backbone network 1. The original training point cloud is then input as training data into 3D U-net backbone network 1. After completing the first round of training, 3D U-net backbone network 1 can output the semantic prediction result and the predicted centroid offset vector. After labeling the first training point cloud with object-oriented tags at multiple levels, the point cloud is reassembled to obtain a reassembled training point cloud. This reassembled training point cloud is used as supervised data, while a reassembled training point cloud without multi-level pseudo-instance labels is used as training data to train the 3D U-net backbone network 2. The trained 3D U-net backbone network 2 can output multi-level predicted point clouds. Core points and edge points can be distinguished using the multi-level object-oriented tags in the multi-level predicted point clouds. The multi-level predicted point clouds are moved using centroid offset vectors to obtain influence point clouds. Core points in the influence point clouds are clustered to obtain core point clouds corresponding to the multi-level predicted point clouds. Influence maps are generated based on the influence values ​​between core points and other points in the influence point clouds. Edge points are then absorbed into the corresponding core point clouds, finally completing instance segmentation and obtaining the third training point cloud. Finally, the third training point cloud is used to replace the first training point cloud as the supervision data for the 3D U-net backbone network 1, and the 3D U-net backbone network 1 is trained in the second round to obtain the trained 3D U-net backbone network 1.

[0284] based on Figure 17 The instance splitting method in the middle, Figure 18A The initial pseudo-instance point cloud is shown. It can be seen that the point clouds of three chairs around the table are connected and predicted as a single instance. After filtering the initial pseudo-instance, these three chairs can be filtered out, resulting in... Figure 18B The first training point cloud; based on the first training point cloud, the first backbone network predicts the instance segmentation result, such as... Figure 18C As shown, two out of three chairs were predicted as one instance, indicating an improvement in instance segmentation performance. Figure 18D It is a multi-level predicted point cloud generated by the second backbone network; Figure 18E Based on Figure 18D The results of core point clustering; Figure 18F Based on Figure 18DThe instance segmentation results after edge point absorption of the core point cloud in the image show that... Figure 18D The three chairs are well distinguished, compared to Figure 18C The instance segmentation effect is better.

[0285] based on Figure 17 The instance segmentation network in the dataset, 3D U-net, uses a 7-layer U-shaped network with each supervoxel being 2cm in size. The batch size of the training data for 3D U-net1 is set to 8, and the batch size of the training data for 3D U-net2 is set to 16. The reconstructed training point cloud consists of 5000 virtual scenes, with an initial learning rate of 0.001. For the asymmetric inference algorithm, the hyperparameter is set to 3. The search radius of the BFS algorithm for obtaining the initial pseudo-instance point cloud from the original semantic point cloud is 5cm. The preset edge levels are 0 and 1, and the preset core levels are 2-4. The training set uses ScanNet-v2, and three different average precision values ​​of each pseudo-instance label in the third training point cloud are obtained, as shown in Table 1.

[0286] Table 1

[0287]

[0288] As can be seen from Table 1, for objects in different indoor scenes, the average score of each indicator exceeds 75 points. Here, mAP represents the score range between 50% and 95%, with a step size of 5%. The average scores of mAP@0.5 and mAP@0.25 are close to 85. In other words, the instance segmentation method provided in this embodiment of the present disclosure has a very good effect on segmenting instances.

[0289] As shown in Table 2, the validation set uses ScanNet-v2. Different instance segmentation methods are used to segment the validation set to obtain the average accuracy of different methods.

[0290] Table 2

[0291]

[0292] As can be seen from Table 2, the average accuracy of the fully supervised method (PointGroup) and the instance segmentation method (AOIA) provided in this embodiment is basically the same across different evaluation metrics. Based on the instance segmentation method provided in this embodiment, the semantic prediction results are smoothed using average pooling. Among different evaluation metrics, the three different AP values ​​are higher than AOIA; for weakly supervised methods, such as the four methods of CSC, the sparser the labeled points, the higher the AP. 50 The lower the value, the more points are labeled in the training data, and the higher the AP. SegGroup, however, labels only one point per instance, resulting in a lower AP.50 The value is basically the same as the CSC-100 method. SPIB annotates the 3D bounding box of each instance, and the resulting AP value is lower than that of SegGroup.

[0293] The validation set uses ScanNet-v2. Table 3 shows the instance segmentation network trained with the first training point cloud after filtering and trained with the initial pseudo instance point cloud without filtering. Comparing the different AP values ​​in the three methods, it can be seen that the AP values ​​of the three methods using the first training point cloud are all higher than those using the initial pseudo instance point cloud. In other words, filtering the initial pseudo instance point cloud can improve the instance segmentation effect.

[0294] Table 3

[0295] Training point cloud AP <![CDATA[AP 50 ]]> <![CDATA[AP 25 ]]> Initial pseudo-instance point cloud 28.7 46.4 62.3 First training point cloud 30.3 48.5 65.5

[0296] The validation set uses ScanNet-v2. Table 4 shows the impact of different preset core level settings on instance segmentation, the impact of using centroid regression, and the impact of using projection. Table 4 shows that preset core levels 1-4, i.e., 80% of points are set as core points, improve all three AP values ​​compared to settings of 40% and 60%, and all are higher than the baseline method of instance segmentation using BFS search. Furthermore, the AP value using centroid regression is always higher than the AP value without centroid regression. This embodiment uses centroid regression to move the point cloud towards the prediction center to generate an influence point cloud. In the first round of training of 3D U-net1, using projection and ignoring depth coordinates (i.e., the z-coordinate), the three AP values ​​are significantly improved compared to not using projection.

[0297] Table 4

[0298] Instance splitting method Centroid Regression Core Point Ratio projection AP <![CDATA[AP 50 ]]> <![CDATA[AP 25 ]]> Baseline BFS 100％ 62 73.5 76.9 Baseline BFS √ 100％ 67.9 78.2 82.1 AOIA (Ours) √ Top 40% 68.5 78.8 82.9 AOIA (Ours) √ Top 60% 69.6 80 83.7 AOIA (Ours) √ Top 80% 70.9 80.8 83.9 AOIA (Ours) √ Top 80% √ 75.6 83.9 86.3

[0299] Figure 19A Multi-level point clouds are shown, based on Figure 19A Multi-level point clouds in Figure 19B The diagram illustrates the effect of using 40% of the points for core point clustering. Figure 19C The diagram illustrates the effect of using 80% of the points as core points for core point clustering.

[0300] It should be noted that the smaller the percentage of core points, the better the instance segmentation, but it is also easy to cause oversegmentation. Therefore, an appropriate percentage needs to be used to achieve the best results.

[0301] Based on the foregoing embodiments, this disclosure provides an instance segmentation device, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0302] Figure 20 This is a schematic diagram of the composition structure of an instance segmentation device provided in an embodiment of this disclosure, as shown below. Figure 20 As shown, the instance segmentation device 600 includes: a first prediction module 610, a second prediction module 620, a moving module 630, a core clustering module 640, and an edge clustering module 650; wherein:

[0303] The first prediction module 610 is used to perform semantic prediction and centroid offset prediction on the acquired point cloud to be segmented through the first backbone network, so as to obtain the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels.

[0304] The second prediction module 620 is used to perform object-oriented prediction on the point cloud to be segmented through the second backbone network to obtain a multi-level point cloud including multiple levels of object-oriented labels; the multiple levels of object-oriented labels are used to characterize the shape of each instance, as well as internal and external information.

[0305] The moving module 630 is used to move each point in the multi-level point cloud to its corresponding prediction center according to the centroid offset vector, so as to obtain the influence point cloud;

[0306] The core clustering module 640 is used to cluster the core points in the influence point cloud according to the semantic prediction results, so as to obtain at least one core point cloud for each semantic category; the core points are points whose object label level is a preset core level.

[0307] The edge clustering module 650 is used to cluster the edge points in the influence point cloud into the target core point cloud in at least one core point cloud of the corresponding semantic category to obtain the instance segmentation result; the edge points are points whose object label level is a preset edge level.

[0308] In some embodiments, the edge clustering module 650 is further configured to calculate the influence value between each edge point and at least one core point cloud of the corresponding semantic category, thereby obtaining at least one influence value for each edge point; the influence value is used to characterize the average influence value between an edge point and all points in a core influence point cloud of the corresponding semantic category; the core point cloud corresponding to the maximum value among the at least one influence value of each edge point is taken as the target core point cloud corresponding to each edge point; each edge point is clustered into the corresponding target core point cloud to obtain at least one instance segmentation point cloud for each semantic category, which is taken as the instance segmentation result.

[0309] In some embodiments, the instance segmentation device 600 further includes: a training module; the training module is further configured to: perform semantic prediction and centroid offset prediction on the point cloud to be segmented through a first backbone network; before obtaining the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented, cluster the original semantic point cloud to obtain an initial pseudo-instance point cloud; filter the initial pseudo-instance point cloud according to the number of points contained in each pseudo-instance in the initial pseudo-instance point cloud to obtain a first training point cloud; train an initial second backbone network based on the first training point cloud to obtain a second backbone network; and train an initial first backbone network based on the first training point cloud and the original semantic point cloud to obtain the first backbone network.

[0310] In some embodiments, the training module is further configured to sort the number of points in each pseudo-instance point cloud of each semantic category to obtain the number order of each semantic category; select pseudo-instance point clouds within a preset number order range from the number order of each semantic category to obtain the training point cloud of each semantic category, wherein the first training point cloud includes the training point cloud of each semantic category.

[0311] In some embodiments, the training module is further configured to compress the first training point cloud to obtain the first compressed point cloud; determine a multi-level training point cloud corresponding to each pseudo-instance point cloud based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud; the multi-level training point cloud includes the multiple levels of object labels; perform point cloud recombination based on the multi-level training point cloud corresponding to each pseudo-instance point cloud to obtain a recombined training point cloud; and train the initial second backbone network using the recombined training point cloud to obtain the second backbone network and the multi-level predicted point cloud output by the second backbone network.

[0312] In some embodiments, the training module is further configured to calculate the distance between each point in each pseudo-instance point cloud and each point in the corresponding compressed pseudo-instance point cloud, thereby obtaining multiple distances corresponding to each point in each pseudo-instance point cloud; take the minimum distance among the multiple compressed distances corresponding to each point in each pseudo-instance point cloud as the compressed distance between each point in each pseudo-instance point cloud and the corresponding compressed point cloud, thereby obtaining multiple compressed distances for each pseudo-instance point cloud; and determine the multiple levels of object labels for each pseudo-instance point cloud based on the multiple compressed distances for each pseudo-instance point cloud, thereby obtaining the multi-level training point cloud corresponding to each pseudo-instance point cloud.

[0313] In some embodiments, the training module is further configured to perform point cloud recombination on the multi-level training point cloud according to the preset point cloud attributes in the recombination template to obtain the recombined training point cloud; the preset point cloud attributes include: preset point cloud quantity, preset point cloud volume, and preset point cloud position; there are no intersecting regions between the preset point cloud positions.

[0314] In some embodiments, the training module is further configured to obtain the (k-1)th updated second backbone network; where k is a positive integer; when k equals 1, the (k-1)th updated second backbone network is the initial second backbone network; extract features from each point of the reconstructed training point cloud using the (k-1)th updated second backbone network to obtain the kth second point cloud feature of each point; perform a second linear transformation on the kth second point cloud feature of each point to obtain the kth second linear transformation feature; the dimension of the kth second linear transformation feature is equal to the dimension of the plurality of equal... The number of object labels at each level corresponds one-to-one; the k-th second cross-entropy between the k-th second linear transformation feature of each point and the corresponding real object label is calculated, and the mean of the k-th second cross-entropy of all points is used as the k-th object loss; if the difference between the k-th object loss and the (k-1)-th object loss is continuously less than or equal to the second preset difference threshold, and the number of consecutive differences is greater than or equal to the second preset number, training is stopped, and the (k-1)-th updated second backbone network is used as the second backbone network.

[0315] In some embodiments, the training module is further configured to calculate the second target dimension value in the k-th second linear transformation feature of each point, and the k-th second cross-entropy between the corresponding target level training label value, and obtain the mean of the k-th second cross-entropy of all points as the k-th objectivity loss, and then, if the difference of the k-th objectivity loss is greater than a preset difference threshold, or if the difference of the k-th objectivity loss is continuously less than or equal to a second difference threshold, and the number of consecutive occurrences is less than a second preset number, update the (k-1)-th second backbone network according to the k-th objectivity loss to obtain the k-th updated second backbone network.

[0316] In some embodiments, the training module is further configured to train the initial first backbone network based on the pseudo-instance labels in the first training point cloud and the corresponding semantic labels in the original semantic point cloud to obtain an initial third backbone network, a third centroid offset vector, and a third semantic prediction result; perform instance segmentation on the multi-level prediction point cloud output by the second backbone network based on the third semantic prediction result and the third centroid offset vector to obtain a third training point cloud; and train the initial third backbone network based on the pseudo-instance labels in the third training point cloud and the semantic labels in the original semantic point cloud to obtain the first backbone network.

[0317] In some embodiments, the training module is further configured to obtain the first backbone network updated for the (α-1)th time; where α is a positive integer; when α equals 1, the first backbone network updated for the (α-1)th time is the initial first backbone network; based on the first backbone network updated for the (α-1)th time, the semantic labels in the original semantic point cloud, and the pseudo-instance labels in the first training point cloud, the first semantic prediction result for the (α-1)th time, the first semantic loss for the (α-1)th time, and the first prediction offset vector for the (α-1)th time are obtained; the first offset loss for the (α-1)th time is used to characterize the difference between the centroid of the instance predicted by the first backbone network updated for the (α-1)th time and the centroid of the real pseudo-instance; the first training point cloud contains the real pseudo-instances; and point prediction is performed based on the first semantic prediction result for the (α-1)th time and the original semantic point cloud. Cloud clustering is performed to obtain the first instance prediction result of the αth time; based on the first instance prediction result of the αth time, the first training point cloud, and the first prediction offset vector of the αth time, a first offset loss is determined; the sum of the first semantic loss of the αth time and the first offset loss of the αth time is taken as the first loss of the αth time; if the difference between the first loss of the αth time and the first loss of the α-1th time is continuously less than or equal to a first difference threshold, and the number of consecutive differences is greater than or equal to a first preset number, training is stopped, and the first backbone network updated in the α-1th time is taken as the initial third backbone network; and the first prediction offset vector of the αth time is taken as the third centroid offset vector; the first semantic prediction result of the αth time is taken as the third semantic prediction result.

[0318] In some embodiments, the training module is further configured to: perform an α-th feature extraction on each original point in the original training point cloud through the first backbone network updated for the (α-1)th time, to obtain an α-th first point cloud feature for each original point; perform semantic prediction on the α-th first point cloud feature of each original point through the first backbone network updated for the (α-1)th time, to obtain an α-th first semantic prediction result; determine the α-th first semantic loss based on the α-th first point cloud feature of each original point and the semantic label of the corresponding original point; and perform feature transformation on the α-th first point cloud feature of each original point to obtain an α-th first predicted centroid offset vector for each original point.

[0319] In some embodiments, the training module is further configured to determine the average coordinates of all points in each instance in the αth first instance prediction result as the αth first instance centroid vector of each point in the corresponding instance; subtract the coordinates of the corresponding point from the αth first instance centroid vector of each point corresponding to the first training point cloud to obtain the αth first vector difference of each point corresponding to the first training point cloud, which is used as the αth first true centroid offset vector of each point corresponding to the first training point cloud; and determine the αth first offset loss based on the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point.

[0320] In some embodiments, the training module is further configured to perform semantic prediction on the first point cloud features of each original point through the first backbone network updated for the (α-1)th time, to obtain the initial semantic prediction point cloud for the αth time; and to smooth the semantic feature vector in each supervoxel of the initial semantic prediction point cloud for the αth time through average pooling, to obtain the fine-tuned initial semantic prediction point cloud for the αth time as the result of the first semantic prediction for the αth time.

[0321] In some embodiments, the training module is further configured to: calculate the norm of the difference between the α-th first predicted centroid offset vector of each point corresponding to the first training point cloud and the α-th first true centroid offset vector of the corresponding point, to obtain the α-th first difference norm of each point corresponding to the first training point cloud; average the α-th first difference norm of each point corresponding to the first training point cloud to obtain the α-th first sub-offset loss; calculate the ratio between the norm of the α-th first predicted centroid offset vector of each point corresponding to the first training point cloud and the α-th first predicted centroid offset vector of the corresponding point, as the α-th first prediction ratio; and calculate the... The ratio between the first true centroid offset vector of the αth time for each point corresponding to the first training point cloud and the norm of the first true centroid offset vector of the αth time is taken as the first actual ratio of the αth time. The product of the first prediction ratio of the αth time for each point corresponding to the first training point cloud and the first actual ratio of the corresponding point is determined to obtain the first direction product of the αth time for each point corresponding to the first training point cloud. The average value of the first direction product of the αth time for each point corresponding to the first training point cloud is calculated to obtain the second sub-offset loss of the αth time. The first offset loss of the αth time is obtained by subtracting the second sub-offset loss of the αth time from the first sub-offset loss of the αth time.

[0322] In some embodiments, the training module is further configured to, after taking the sum of the αth first semantic loss and the αth first offset loss as the αth first loss, perform the αth update on the α-1th updated first backbone network according to the αth first loss when the αth first loss difference is greater than the first loss difference threshold, or when the αth first loss difference is continuously less than or equal to the first loss difference threshold and the number of consecutive occurrences is less than the first preset number, to obtain the αth updated first backbone network.

[0323] In some embodiments, the training module is further configured to move each point in the multi-level prediction point cloud to its corresponding prediction center according to the third centroid offset vector to obtain an influence prediction point cloud; cluster the core points in the influence prediction point cloud according to the third semantic prediction result to obtain at least one third core point cloud for each semantic category; and cluster the edge points in the influence prediction point cloud into the third target core point cloud in at least one third core point cloud of the corresponding semantic category to obtain the third training point cloud.

[0324] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0325] It should be noted that, in the embodiments of this disclosure, if the above-described instance segmentation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0326] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0327] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0328] This disclosure provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0329] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0330] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0331] It should be noted that, Figure 21 This is a schematic diagram of a hardware entity of the instance segmentation device in an embodiment of this disclosure, such as... Figure 21 As shown, the hardware entity of the instance segmentation device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0332] Processor 701 typically controls the overall operation of computer device 700.

[0333] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0334] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0335] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0336] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0337] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0338] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0339] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0340] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0341] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0342] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An instance segmentation method, characterized in that, include: The first backbone network performs semantic prediction and centroid offset prediction on the acquired point cloud to be segmented, thereby obtaining the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels. The object-oriented prediction of the point cloud to be segmented is performed by the second backbone network to obtain a multi-level point cloud including multiple levels of object-oriented labels; the multiple levels of object-oriented labels are used to characterize the shape of each instance, as well as internal and external information; Based on the centroid offset vector, each point in the multi-level point cloud is moved to its corresponding prediction center to obtain the influence point cloud; Based on the semantic prediction results, the core points in the influence point cloud are clustered to obtain at least one core point cloud for each semantic category; the core points are points whose object label level is a preset core level. The edge points in the influence point cloud are clustered into the target core point cloud in at least one core point cloud of the corresponding semantic category to obtain the instance segmentation result; The edge points are points whose object-specific labels have a preset edge level; The second backbone network, trained based on the first training point cloud, includes: The first training point cloud is compressed to obtain the first compressed point cloud; Based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud, a multi-level training point cloud corresponding to each pseudo-instance point cloud is determined; the multi-level training point cloud includes the multiple levels of object labels. Based on the multi-level training point cloud corresponding to each pseudo-instance point cloud, point cloud recombination is performed to obtain the recombined training point cloud. The initial second backbone network is trained using the recombined training point cloud to obtain the second backbone network and the multi-level predicted point cloud output by the second backbone network.

2. The method according to claim 1, characterized in that, The step of clustering edge points in the influence point cloud into target core point clouds in at least one core point cloud of the corresponding semantic category to obtain instance segmentation results includes: Calculate the influence value between each edge point and at least one core point cloud of the corresponding semantic category to obtain at least one influence value for each edge point; the influence value is used to characterize the average influence value between an edge point and all points in a core influence point cloud of the corresponding semantic category. The core point cloud corresponding to the maximum value of at least one influence value of each edge point is taken as the target core point cloud corresponding to each edge point. Each edge point is clustered into the corresponding target core point cloud to obtain at least one instance segmentation point cloud for each semantic category, which is used as the instance segmentation result.

3. The method according to claim 1 or 2, characterized in that, Before performing semantic prediction and centroid offset prediction on the point cloud to be segmented through the first backbone network to obtain the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented, the method further includes: Clustering the original semantic point cloud yields an initial pseudo-instance point cloud; The initial pseudo-instance point cloud is filtered based on the number of points contained in each pseudo-instance to obtain the first training point cloud. The initial second backbone network is trained based on the first training point cloud to obtain the second backbone network; The initial first backbone network is trained based on the first training point cloud and the original semantic point cloud to obtain the first backbone network.

4. The method according to claim 3, characterized in that, The step of filtering the initial pseudo-instance point cloud based on the number of points in each pseudo-instance in the initial pseudo-instance point cloud to obtain the first training point cloud includes: Sort the number of points in the point cloud of each pseudo-instance for each semantic category to obtain the order of the number of points for each semantic category; From the quantity order of each semantic category, select pseudo-instance point clouds within a preset quantity order range to obtain the training point cloud of each semantic category. The first training point cloud includes the training point cloud of each semantic category.

5. The method according to claim 1, characterized in that, The step of determining the multi-level training point cloud corresponding to each pseudo-instance point cloud based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud includes: Calculate the distance between each point in each pseudo-instance point cloud and each point in the corresponding compressed pseudo-instance point cloud to obtain multiple distances corresponding to each point in each pseudo-instance point cloud; The minimum distance among the multiple distances corresponding to each point in each pseudo-instance point cloud is taken as the compression distance between each point in each pseudo-instance point cloud and the corresponding compressed point cloud, thus obtaining multiple compression distances for each pseudo-instance point cloud. Based on the multiple compression distances of each pseudo-instance point cloud, the multiple levels of object-oriented labels for each pseudo-instance point cloud are determined, thereby obtaining the multi-level training point cloud corresponding to each pseudo-instance point cloud.

6. The method according to claim 1, characterized in that, The step of reconstructing the point cloud based on the multi-level training point cloud corresponding to each pseudo-instance point cloud to obtain the reconstructed training point cloud includes: According to the preset point cloud attributes in the recombination template, the multi-level training point cloud is recombined to obtain the recombined training point cloud; the preset point cloud attributes include: preset point cloud quantity, preset point cloud volume, and preset point cloud position; there are no intersecting regions between the preset point cloud positions.

7. The method according to claim 1, characterized in that, The step of training the initial second backbone network using the reconstructed training point cloud to obtain the second backbone network includes: Obtain the second backbone network updated for the (k-1)th time; where k is a positive integer; when k equals 1, the second backbone network updated for the (k-1)th time is the initial second backbone network; The second backbone network updated in the (k-1)th iteration is used to extract features from each point of the recombined training point cloud, resulting in the second point cloud features of each point in the kth iteration. A second linear transformation is performed on the k-th second point cloud feature of each point to obtain the k-th second linear transformation feature; the dimension of the k-th second linear transformation feature corresponds one-to-one with the number of levels of the multiple levels of object-oriented labels; Calculate the kth second cross-entropy between the kth second linear transformation feature of each point and the corresponding point's true objectivity label, and obtain the mean of the kth second cross-entropy of all points as the kth objectivity loss; If the difference between the k-th object loss and the (k-1)-th object loss is continuously less than or equal to the second preset difference threshold, and the number of consecutive differences is greater than or equal to the second preset number, training is stopped, and the second backbone network updated in the (k-1)-th iteration is used as the second backbone network.

8. The method according to claim 7, characterized in that, After calculating the k-th second target dimension value in the k-th second linear transformation feature of each point and the k-th second cross-entropy between it and the corresponding target level training label value, and obtaining the mean of the k-th second cross-entropy of all points as the k-th objectivity loss, the method further includes: If the difference in the object-oriented loss of the kth time is greater than a preset difference threshold, or if the difference in the object-oriented loss of the kth time is continuously less than or equal to a second difference threshold, and the number of consecutive occurrences is less than a second preset number, the second backbone network of the (k-1)th time is updated for the kth time based on the object-oriented loss of the kth time, and the second backbone network updated for the kth time is obtained.

9. The method according to claim 3, characterized in that, The process of training an initial first backbone network based on the first training point cloud and the original semantic point cloud to obtain the first backbone network includes: The initial first backbone network is trained based on the pseudo-instance labels in the first training point cloud and the corresponding semantic labels in the original semantic point cloud to obtain the initial third backbone network, the third centroid offset vector, and the third semantic prediction result. Based on the third semantic prediction result and the third centroid offset vector, the multi-level prediction point cloud output by the second backbone network is segmented into instances to obtain the third training point cloud; The initial third backbone network is trained based on the pseudo-instance labels in the third training point cloud and the semantic labels in the original semantic point cloud to obtain the first backbone network.

10. The method according to claim 9, characterized in that, The initial first backbone network is trained based on the pseudo-instance labels in the first training point cloud and the corresponding semantic labels in the original semantic point cloud to obtain an initial third backbone network, a third centroid offset vector, and a third semantic prediction result, including: Obtain the first backbone network updated for the (α-1)th time; where α is a positive integer; when α equals 1, the first backbone network updated for the (α-1)th time is the initial first backbone network; Based on the first backbone network updated in the (α-1)th time, the semantic labels in the original semantic point cloud, and the pseudo-instance labels in the first training point cloud, the first semantic prediction result, the first semantic loss, and the first prediction offset vector are obtained in the (α-1)th time. The first offset loss is used to characterize the difference between the centroid of the instance predicted by the first backbone network updated in the (α-1)th time and the centroid of the real pseudo-instance. The first training point cloud contains the real pseudo-instances. Based on the first semantic prediction result of the αth time and the original semantic point cloud, point cloud clustering is performed to obtain the first instance prediction result of the αth time. The first offset loss is determined based on the first instance prediction result of the αth time, the first training point cloud, and the first prediction offset vector of the αth time. The sum of the αth first semantic loss and the αth first offset loss is taken as the αth first loss; If the difference between the first loss in the αth iteration and the first loss in the (α-1th iteration)th iteration is continuously less than or equal to a first difference threshold, and the number of consecutive occurrences is greater than or equal to a first preset number, training is stopped, and the first backbone network updated in the (α-1th iteration)th iteration is obtained as the initial third backbone network; and the first prediction offset vector in the αth iteration is obtained as the third centroid offset vector; and the first semantic prediction result in the αth iteration is obtained as the third semantic prediction result.

11. The method according to claim 10, characterized in that, The process of obtaining the first semantic prediction result, the first semantic loss, and the first prediction offset vector based on the first backbone network updated in the (α-1)th time, the semantic labels in the original semantic point cloud, and the pseudo-instance labels in the first training point cloud includes: The first backbone network updated for the α-1th time performs the α-th feature extraction on each original point in the original training point cloud to obtain the α-th first point cloud feature of each original point. The semantic prediction of the first point cloud features of each original point is performed using the first backbone network updated in the (α-1)th time, to obtain the first semantic prediction result of the αth time; and, Based on the α-th first point cloud features of each original point and the semantic label of the corresponding original point, the α-th first semantic loss is determined; and, The first point cloud feature of each original point is transformed to obtain the first predicted centroid offset vector of each original point.

12. The method according to claim 10, characterized in that, The step of determining the first offset loss based on the first instance prediction result of the αth time, the first training point cloud, and the first prediction offset vector of the αth time includes: The average coordinates of all points in each instance in the αth first instance prediction result are determined as the centroid vector of each point in the corresponding instance in the αth first instance. By subtracting the coordinates of the corresponding point from the first instance centroid vector of each point corresponding to the first training point cloud, the first vector difference of the first instance corresponding to each point in the first training point cloud is obtained, which is used as the first true centroid offset vector of the first instance corresponding to each point in the first training point cloud. The first offset loss for the αth time is determined based on the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point.

13. The method according to claim 11, characterized in that, The step of performing semantic prediction on the first point cloud features of each original point through the first backbone network updated in the (α-1)th time to obtain the first semantic prediction result in the αth time includes: The semantic prediction of the first point cloud features of each original point is performed on the first backbone network updated in the (α-1)th time to obtain the initial semantic prediction point cloud in the αth time. The semantic feature vector in each supervoxel of the α-th initial semantic prediction point cloud is smoothed by average pooling to obtain the fine-tuned α-th initial semantic prediction point cloud as the α-th first semantic prediction result.

14. The method according to claim 12, characterized in that, The step of determining the first offset loss for the αth time based on the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point includes: Calculate the norm of the difference between the αth first predicted centroid offset vector of each point corresponding to the first training point cloud and the αth first true centroid offset vector of the corresponding point, and obtain the αth first difference norm of each point corresponding to the first training point cloud. The first sub-offset loss of the αth time is obtained by averaging the first difference norm of each point corresponding to the first training point cloud; Calculate the ratio between the first predicted centroid offset vector of the αth time for each point corresponding to the first training point cloud and the norm of the first predicted centroid offset vector of the corresponding point, as the first predicted ratio of the αth time; and calculate the ratio between the first true centroid offset vector of the αth time for each point corresponding to the first training point cloud and the norm of the first true centroid offset vector of the αth time, as the first true ratio of the αth time. The product of the αth first prediction ratio of each point corresponding to the first training point cloud and the αth first actual ratio of the corresponding point is determined to obtain the αth first direction product of each point corresponding to the first training point cloud. The average value of the first direction product of each point corresponding to the first training point cloud is calculated to obtain the second sub-offset loss of the αth time. The first offset loss of the αth time is obtained by subtracting the second sub-offset loss of the αth time from the first sub-offset loss of the αth time.

15. The method according to any one of claims 10-14, characterized in that, After taking the sum of the α-th first semantic loss and the α-th first offset loss as the α-th first loss, the method further includes: If the first loss difference in the αth iteration is greater than the first loss difference threshold, or if the first loss difference in the αth iteration is continuously less than or equal to the first loss difference threshold and the number of consecutive iterations is less than the first preset number, the first backbone network updated in the α-1th iteration is updated in the αth iteration based on the first loss in the αth iteration, and the first backbone network updated in the αth iteration is obtained.

16. The method according to claim 9, characterized in that, The step of performing instance segmentation on the multi-level predicted point cloud output by the second backbone network based on the third semantic prediction result and the third centroid offset vector to obtain the third training point cloud includes: Based on the third centroid offset vector, each point in the multi-level prediction point cloud is moved to its corresponding prediction center to obtain the influence prediction point cloud; Based on the third semantic prediction result, the core points in the influence prediction point cloud are clustered to obtain at least one third core point cloud for each semantic category. The edge points in the influence prediction point cloud are clustered into the third target core point cloud in at least one third core point cloud of the corresponding semantic category to obtain the third training point cloud.

17. An instance segmentation device, characterized in that, include: Memory, used to store computer programs; processor; The processor is used to execute the computer program to: The first backbone network performs semantic prediction and centroid offset prediction on the acquired point cloud to be segmented, thereby obtaining the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels. The object-oriented prediction of the point cloud to be segmented is performed by the second backbone network to obtain a multi-level point cloud including multiple levels of object-oriented labels; the multiple levels of object-oriented labels are used to characterize the shape of each instance, as well as internal and external information; Based on the centroid offset vector, each point in the multi-level point cloud is moved to its corresponding prediction center to obtain the influence point cloud; Based on the semantic prediction results, the core points in the influence point cloud are clustered to obtain at least one core point cloud for each semantic category; the core points are points whose object label level is a preset core level. The edge points in the influence point cloud are clustered into the target core point cloud in at least one core point cloud of the corresponding semantic category to obtain the instance segmentation result; the edge points are points whose object label level is a preset edge level. The second backbone network, trained based on the first training point cloud, includes: The first training point cloud is compressed to obtain the first compressed point cloud; Based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud, a multi-level training point cloud corresponding to each pseudo-instance point cloud is determined; the multi-level training point cloud includes the multiple levels of object labels. Based on the multi-level training point cloud corresponding to each pseudo-instance point cloud, point cloud recombination is performed to obtain the recombined training point cloud. The initial second backbone network is trained using the recombined training point cloud to obtain the second backbone network and the multi-level predicted point cloud output by the second backbone network.

18. A computer-readable storage medium, characterized in that, The computer program is stored therein, wherein when the computer program is executed by a processor, the processor causes the processor to: The first backbone network performs semantic prediction and centroid offset prediction on the acquired point cloud to be segmented, thereby obtaining the semantic prediction result and centroid offset vector of each point in the point cloud to be segmented; the first backbone network and the second backbone network are trained based on the first training point cloud; the first training point cloud is determined by the original semantic point cloud with semantic labels. The object-oriented prediction of the point cloud to be segmented is performed by the second backbone network to obtain a multi-level point cloud including multiple levels of object-oriented labels; the multiple levels of object-oriented labels are used to characterize the shape of each instance, as well as internal and external information; Based on the centroid offset vector, each point in the multi-level point cloud is moved to its corresponding prediction center to obtain the influence point cloud; Based on the semantic prediction results, the core points in the influence point cloud are clustered to obtain at least one core point cloud for each semantic category; the core points are points whose object label level is a preset core level. The edge points in the influence point cloud are clustered into the target core point cloud in at least one core point cloud of the corresponding semantic category to obtain the instance segmentation result; the edge points are points whose object label level is a preset edge level. The second backbone network, trained based on the first training point cloud, includes: The first training point cloud is compressed to obtain the first compressed point cloud; Based on the compression distance between each point in each pseudo-instance point cloud in the first training point cloud and the corresponding compressed pseudo-instance point cloud in the first compressed point cloud, a multi-level training point cloud corresponding to each pseudo-instance point cloud is determined; the multi-level training point cloud includes the multiple levels of object labels. Based on the multi-level training point cloud corresponding to each pseudo-instance point cloud, point cloud recombination is performed to obtain the recombined training point cloud. The initial second backbone network is trained using the recombined training point cloud to obtain the second backbone network and the multi-level predicted point cloud output by the second backbone network.

Images