A transformer-based 3D instance segmentation method, system, and device

Abstract

The application discloses a kind of 3D instance segmentation methods, systems and equipment based on Transformer, belong to computer vision field, including: the feature of seed point is aggregated by cross attention, self-attention and feedforward neural network;Wherein seed point is obtained by carrying out the farthest distance sampling to input point cloud, and the feature of seed point is the multistage feature extracted to input point cloud by 3D-UNet of sparse convolution;In order to solve the problem existing in query sampling, by query initialization module, to ensure the high coverage and low repetition rate of object instance.In addition, we designed associated Transformer decoder to suppress the interference of noisy background query to obtain better instance segmentation result.Through the design of this patent, to a large extent, the problem of unreasonable query distribution and noise background query interference in 3D instance segmentation is solved, and finally helps the model to accurately and efficiently segment object instance and identify object category.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a 3D instance segmentation method, system, and device based on Transformer. Background Technology

[0002] 3D instance segmentation is a fundamental task in 3D scene understanding, aiming to predict the semantic label and binary foreground mask for each object in the scene. With the popularization of AR / VR, 3D indoor scanning, and autonomous driving, 3D instance segmentation has become a key technology for promoting scene understanding.

[0003] Currently, 3D instance segmentation methods can be broadly categorized into three types: proposal-based, grouping-based, and query-based methods. Query-based methods are considered a class of Transformer-based query-oriented methods, where each object instance is represented as an instance query. Query-oriented methods require distributing a large number of queries across the entire scene to cover most of the foreground objects, so that each covered object instance has one or more queries. The Transformer decoder then learns the instance queries by iteratively focusing on multi-layer point cloud features. Finally, the instance queries aggregate the point cloud features to generate masks for all instances in parallel. Therefore, the distribution of queries has a significant impact on the instance segmentation results. Existing query-based methods suffer from poor coverage and repetition rates in query distribution, hence this proposal suggests a Transformer-based 3D instance segmentation method. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a Transformer-based 3D instance segmentation method to optimize the query initialization process and achieve a query distribution with high coverage and low repetition rate. To suppress the interference of noisy background queries on the segmentation results, this patent also designs a denoising module with contrast loss to exclude noisy background queries from instance queries.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] Firstly, this application proposes a 3D instance segmentation method based on Transformer, including:

[0007] Obtain multi-level features and seed points from the input point cloud;

[0008] Features of aggregated seed points; the aggregated seed points are filtered using translation operations and non-maximum suppression operations guided by intersection-union ratio to obtain the query points;

[0009] Multi-level features are divided into disjoint sets based on the similarity between multi-level features and the features of the query point;

[0010] Cluster the query points and their corresponding sets to obtain the query output;

[0011] The query output is input into a multilayer perceptron, which outputs the instance segmentation result.

[0012] In some embodiments, the query point is encoded with the real centroid to obtain a de-interferenced query point; the output features of the query are determined based on the de-interferenced query point.

[0013] Among them, the true center point is obtained by resampling the center point of the instance from the true annotation, satisfying: Q gt ~N(C gt , σ 2 )

[0014] Where N represents a Gaussian distribution, σ represents the bias, and C gt This represents the mean.

[0015] In some embodiments, determining the output features of the query based on the de-interference query point includes the following steps:

[0016] The query output is matched with the ground truth labels in the dataset. Queries with minimum matching cost are paired with perturbed ground truth centroids to form positive sample pairs. Positive sample queries are encouraged to have output features consistent with the output features of their corresponding centroids. Noisy background queries that do not match are removed from instances based on contrastive loss, which satisfies the following conditions:

[0017]

[0018] Where d(.,.) is the distance metric function, ε is the temperature parameter in contrastive learning; K is the number of queries; N gt q represents the number of true annotations, β represents the number of true centroids of the perturbations generated by true annotation j; the perturbation centroids of query i and true annotation j form a positive sample pair. i The characteristics representing query i The feature representing the m-th perturbation point of the true annotation j.

[0019] In some embodiments, the characteristics of the aggregation seed point include the following steps:

[0020] Based on Mask3D, the original point cloud is divided into multiple superpoints. Then, average pooling is performed on the last-level feature F2 in the multi-level feature set to obtain the superpoint feature F. super ;

[0021] For the superpoint feature F super and seed point features F seed Perform cross attention.

[0022]

[0023] Where Q = F seed W q K = F super W k V = F super W v C is the number of feature channels, W q W k W v It is a linear mapping layer;

[0024] After updating the seed point features using cross-attention, self-attention is used to update the seed point features:

[0025]

[0026] Where Q' = F' seed W q K' = F' seed W k V' = F' seed W v ;

[0027] After the self-attention update, a feedforward neural network composed of multilayer perceptrons is used to update the features of the seed points.

[0028] In some embodiments, the step of filtering the aggregated seed points through translation and intersection-union ratio-guided non-maximum suppression operations to obtain query points includes the following steps:

[0029] Predict each seed point P using a multilayer perceptron. seed Relative to its matching instance The coordinate offset of the center point; after obtaining this coordinate offset, add it to the original coordinate p, as follows:

[0030] p = MLP(s) + p, where s refers to the features;

[0031] The seed point P is obtained by calculating the dot product between the superpoint feature and the seed point feature. seed Corresponding instance mask M ins The final mask is obtained by applying a threshold of 0.5 to the output of the Sigmoid function, as shown below:

[0032]

[0033] Using the mask M obtained above b Calculate the cross-union ratio (CUNR) and use CUNR-guided nonmaximum suppression to filter out redundant seed points.

[0034] In some embodiments, dividing the multi-level features into disjoint sets based on the similarity between the multi-level features and the features of the query point includes the following steps:

[0035] F is obtained through the arg max operation. l The allocation is as follows:

[0036]

[0037] Where M i,k F is the dot product of query i and the main feature k. l This is a level 1 feature in a multi-level feature set.

[0038] In some embodiments, obtaining the multi-level features and seed points of the input point cloud includes the following steps:

[0039] Multi-level features F0, F1, and F2 are obtained by encoding the position and color information of the input point cloud using a 3D-UNet based on sparse convolution; seed points P are obtained by sampling the input point cloud at the furthest distance. seed .

[0040] Secondly, this application also proposes a Transformer-based 3D instance segmentation system, including:

[0041] Feature aggregation module: Aggregates the features of seed points through cross attention, self attention, and feedforward neural networks; where seed points are obtained by sampling the input point cloud at the farthest distance, and the features of seed points are multi-level features extracted from the input point cloud by 3D-UNet with sparse convolution;

[0042] Set grouping module: Seed points are filtered using translation operations and non-maximum suppression operations guided by intersection-union ratio to obtain query points;

[0043] Clustering layer module: Based on the similarity of multi-level features and the features of the de-interference query points, the multi-level features are divided into disjoint sets; the de-interference query points and their corresponding sets are processed through cross attention, attention and feedforward neural networks to obtain the query output;

[0044] Denoising module: Matches the query output with the real labels in the dataset. Queries with minimum matching cost form positive sample pairs with the perturbed real centroids. It encourages the output features of positive sample queries to be consistent with the output features of the corresponding centroids, and uses contrastive loss to move mismatched noisy background queries away from instances, thus obtaining the output of denoised queries.

[0045] Output module: The output of the denoised query is processed by a multilayer perceptron to output the instance segmentation result.

[0046] Thirdly, this application also proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor, and when the processor loads and executes the computer program, it employs a Transformer-based 3D instance segmentation method as described above.

[0047] Fourthly, this application also proposes a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, employs a Transformer-based 3D instance segmentation method as described above.

[0048] The beneficial effects of this invention are:

[0049] This patent proposes a query-optimized Transformer for 3D instance segmentation. To address issues with query sampling, we designed a query initialization module to ensure high coverage and low repetition rate of object instances. Furthermore, we designed an associative Transformer decoder to suppress interference from noisy background queries, resulting in better instance segmentation results. This patent design largely solves the problems of unreasonable query distribution and interference from noisy background queries in 3D instance segmentation, ultimately helping the model to accurately and efficiently segment object instances and identify object categories. Attached Figure Description

[0050] The invention will now be further described with reference to the accompanying drawings.

[0051] Figure 1 This is a schematic diagram of the overall framework of this application;

[0052] Figure 2 This is a schematic diagram of the query initialization module of this application;

[0053] Figure 3 This is a schematic diagram of the feature aggregation module of this application;

[0054] Figure 4 This is a schematic diagram of the associated Transformer decoder module in this application;

[0055] Figure 5 This is a schematic diagram of the clustering layer module in this application; Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0058] This application proposes a Transformer-based 3D instance segmentation method, including:

[0059] Obtain multi-level features and seed points from the input point cloud;

[0060] Features of aggregated seed points; the aggregated seed points are filtered using translation operations and non-maximum suppression operations guided by intersection-union ratio to obtain the query points;

[0061] Multi-level features are divided into disjoint sets based on the similarity between multi-level features and the features of the query point;

[0062] Cluster the query points and their corresponding sets to obtain the query output;

[0063] The query output is input into a multilayer perceptron, which outputs the instance segmentation result.

[0064] In some embodiments, the query point is encoded with the real centroid to obtain a de-interferenced query point; the output features of the query are determined based on the de-interferenced query point.

[0065] Among them, the true center point is obtained by resampling the center point of the instance from the true annotation, satisfying: Q gt ~N(C gt , σ 2 )

[0066] Where N represents a Gaussian distribution, σ represents the bias, and C gt This represents the mean.

[0067] In some embodiments, determining the output features of the query based on the de-interference query point includes the following steps:

[0068] The query output is matched with the ground truth labels in the dataset. Queries with minimum matching cost are paired with perturbed ground truth centroids to form positive sample pairs. Positive sample queries are encouraged to have output features consistent with the output features of their corresponding centroids. Noisy background queries that do not match are removed from instances based on contrastive loss, which satisfies the following conditions:

[0069]

[0070] Where d(.,.) is the distance metric function, ε is the temperature parameter in contrastive learning; K is the number of queries; N gt q represents the number of true annotations, β represents the number of true centroids of the perturbations generated by true annotation j; the perturbation centroids of query i and true annotation j form a positive sample pair. i The characteristics representing query i The feature representing the m-th perturbation point of the true annotation j.

[0071] In some embodiments, the characteristics of the aggregation seed point include the following steps:

[0072] Based on Mask3D, the original point cloud is divided into multiple superpoints. Then, average pooling is performed on the last-level feature F2 in the multi-level feature set to obtain the superpoint feature F. super ;

[0073] For the superpoint feature F super and seed point features F seed Perform cross attention.

[0074]

[0075] Where Q = F seed W q K = F super W k V = F super W v C is the number of feature channels, W q W k W v It is a linear mapping layer;

[0076] After updating the seed point features using cross-attention, self-attention is used to update the seed point features:

[0077]

[0078] Where Q' = F' seed W q K' = F' seed W k V' = F' seed W v ;

[0079] After the self-attention update, a feedforward neural network composed of multilayer perceptrons is used to update the features of the seed points.

[0080] In some embodiments, the step of filtering the aggregated seed points through translation and intersection-union ratio-guided non-maximum suppression operations to obtain query points includes the following steps:

[0081] Predict each seed point P using a multilayer perceptron. seed Relative to its matching instance The coordinate offset of the center point; after obtaining this coordinate offset, add it to the original coordinate p, as follows:

[0082] p = MLP(s) + p, where s refers to the features;

[0083] The seed point P is obtained by calculating the dot product between the superpoint feature and the seed point feature. seed Corresponding instance mask M ins The final mask is obtained by applying a threshold of 0.5 to the output of the Sigmoid function, as shown below:

[0084]

[0085] Using the mask M obtained above b Calculate the cross-union ratio (CUNR) and use CUNR-guided nonmaximum suppression to filter out redundant seed points.

[0086] In some embodiments, dividing the multi-level features into disjoint sets based on the similarity between the multi-level features and the features of the query point includes the following steps:

[0087] F is obtained through the arg max operation. l The allocation is as follows:

[0088]

[0089] Where M i,k F is the dot product of query i and the main feature k. l This is a level 1 feature in a multi-level feature set.

[0090] In some embodiments, obtaining the multi-level features and seed points of the input point cloud includes the following steps:

[0091] Multi-level features F0, F1, and F2 are obtained by encoding the position and color information of the input point cloud using a 3D-UNet based on sparse convolution; seed points p are obtained by sampling the input point cloud at the furthest distance. seed .

[0092] This application discloses a 3D instance segmentation system based on Transformer. A 3D instance is a semantic label and binary foreground mask of each point in the input point cloud predicted by Transformer. The system includes a query initialization module and an associated Transformer decoder module.

[0093] The overall execution process is as follows: Figure 1 As shown, assume the input point cloud has N points, each containing position (x, y, z) and color (r, g, b) information. First, we use a 3D-UNet based on sparse convolution to encode the position and color information to obtain multi-level features F0, F1, and F2. Next, we use the seed point P obtained by the furthest distance sampling... seed The query points are input into the query initialization module. Through seed point feature aggregation in the feature aggregation module and seed point filtering in the set grouping module, we obtain high-quality query points Q1. Finally, we input the obtained query points into the proposed correlation Transformer decoder module to obtain the final prediction result. The specific details of the different modules are as follows:

[0094] (1) Query initialization module. The query initialization module mainly consists of two parts: feature aggregation module and set grouping module.

[0095] Feature aggregation module: The purpose of the feature aggregation module is to aggregate seed point P. seed The features are primarily aggregated using cross-attention, self-attention, and feedforward neural networks. Specifically, we first divide the original point cloud into superpoints based on Mask3D. Next, we perform average pooling on the final-level feature F2 based on the superpoint division to obtain the superpoint features F. super Next, we will analyze the superpoint feature F. super and seed point features F seed Perform cross attention.

[0096]

[0097] Where Q = F seed W q K = F super W k V = F super W v C is the number of feature channels, W q W k W v This is a linear mapping layer. After updating the seed point features using cross-attention, we will use self-attention to further update the seed point features.

[0098]

[0099] Where Q' = F' seed W q K' = F' seed W k V' = F' seed W v Finally, after updating the seed point features using self-attention, we will use a feedforward neural network composed of multilayer perceptrons to update the seed point features. The updated seed point P seed The data will be input into the set grouping module for seed point filtering to obtain high-quality query points Q1.

[0100] Set Grouping Module: The set grouping module aims to improve coverage and reduce duplication, and includes a translation operation and an intersection-union-ratio (IU-r) guided nonmaximum suppression operation. The translation operation shifts the seed point to the center of the instance to ensure high coverage, so that most foreground instances have a corresponding seed point. The IU-r guide nonmaximum suppression operation filters out duplicate seed points, ensuring that each foreground instance retains only one high-quality seed point, thereby reducing the duplication rate of seed points.

[0101] Specifically, we first use a multilayer perceptron to predict each seed point P. seed Relative to its matching instance The coordinate offset of the center point. After obtaining this coordinate offset, add it to the original coordinate p, where s refers to the feature:

[0102] p = MLP(s) + p.

[0103] In this way, we can not only move some background points to the foreground, but also bring points on the same instance closer together. This will encourage these points to learn similar features and help non-maximum suppression remove redundant seed points. Then, we obtain the seed point P by calculating the dot product between the superpoint features and the seed point features. seed Corresponding instance mask M ins Finally, we obtain the final mask by applying a threshold of 0.5 to the output of the Sigmoid function, as shown below:

[0104]

[0105] Finally, we use the mask M obtained above. b The cross-union ratio (CUNR) is calculated, and non-maximum suppression guided by the CUNR is used to filter out a large number of redundant seed points. Here, the confidence level of the non-maximum suppression is the product of the class score and the CUNR score.

[0106] (2) Associated Transformer decoder module. The associated Transformer decoder module mainly consists of two parts: clustering layer module and denoising module.

[0107] Clustering layer module: The purpose of the clustering layer is to update the features of query point Q1. For example... Figure 5 As shown, this method is based on the idea of ​​clustering, assigning each point in the scene to its most similar query. Specifically, we first calculate a feature F from a multi-level feature set F0, F1, and F2. l and the features F of query point Q1 Q1 The similarity between them. Next, we use the arg max operation to calculate the similarity between F. l The allocation is as follows:

[0108]

[0109] Where M i,r The features i and F of a certain point in query point Q1 are... l The dot product of features r at a certain point in the matrix. Therefore, we cluster all F... l We divide the queries into disjoint sets. Next, we perform a cross-attention operation on the queries and their corresponding sets. In this way, we only need to compute attention maps on some similar features for each query. Finally, we process the query input into a self-attention layer and a feedforward neural network, where the self-attention layer and feedforward neural network have the same structure as in the query initialization module. In this way, we can fully utilize the features F of query point Q1. Q1 It interacts with multi-level features F0, F1, and F2. After the interaction, we will set the feature F of the query point. Q1 Input into a multilayer perceptron and output instance segmentation results.

[0110] Denoising Module: The denoising module assists in the training of the clustering layers and is not used during inference, therefore it does not affect the final result. In the query initialization module, we move background points around the foreground points and offset them towards the instance centers to improve coverage. However, during the offset process, not every point migrates well to the center position due to the different scales and categories of objects. This causes some background points to potentially affect the information aggregation of the foreground points. Therefore, in this module, we handle the interference caused by the offset of background points. We obtain the center points of the instances from the ground truth annotations and resample them, as shown below:

[0111] Q gt ~N(C gt , σ 2 ),

[0112] Where N represents a Gaussian distribution, σ represents the bias, and C gt Let Q represent the mean. We will then define the true center point after perturbation. gt Encode the query point Q1 obtained in the query initialization module and feed it into the associated Transformer decoder.

[0113] Then, we match the query outputs with the ground truth labels in the dataset. Queries with minimum matching costs form positive pairs with the perturbed ground truth centroids. We encourage the output features of positive queries to align with the output features of the corresponding centroids, and we keep mismatched, noisy background queries away from instances to reduce interference with instance mask prediction. Therefore, we use contrastive loss to achieve this. Assume the number of queries is K and the number of ground truth labels is N. gt For each ground truth label j, β ground truth centroids of the perturbations are generated. Query i and the ground truth label j's perturbation centroids form a positive sample pair.

[0114]

[0115] Where d(.,.) is the distance metric function, ε is the temperature parameter in contrastive learning, and q i The characteristics representing query i The feature representing the m-th perturbation point of the true annotation j.

[0116] Feature F at the query point Q1 After interacting with the multi-level features, we will use the features F of the query point. Q1 Input into a multilayer perceptron and output instance segmentation results.

[0117] This invention can be widely applied in systems such as autonomous driving, robotic arm grasping, and augmented reality to accurately segment and recognize objects in point cloud scenes. In implementation, it can be installed as software on front-end devices, robots, and autonomous vehicles to provide real-time object instance segmentation; alternatively, it can be installed on a back-end server to provide large-scale object segmentation and recognition results for 3D point cloud scenes.

[0118] Table 1: Comparison of experimental results on the ScanNetV2 dataset

[0119] method mAP AP@50 AP@25 Mask3D 55.2 73.7 82.9 The method proposed in this patent 56.5 74.2 83.3

[0120] As shown in Table 1, we compared our method with the current state-of-the-art method on the ScanNetV2 dataset. The results show that our method achieved the best performance on mAP, AP@50, and AP@25, validating the effectiveness of our method.

[0121] This application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it employs any of the Transformer-based 3D instance segmentation methods described in the above embodiments.

[0122] The terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server. The terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0123] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0124] The memory can be an internal storage unit of the terminal device, such as a hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. Furthermore, the memory can be a combination of internal storage units and external storage devices of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0125] In this terminal device, any of the Transformer-based 3D instance segmentation methods in the above embodiments can be stored in the terminal device's memory and loaded and executed on the terminal device's processor for convenient use.

[0126] This application also discloses a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, it employs any of the Transformer-based 3D instance segmentation methods described in the above embodiments.

[0127] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0128] In this computer-readable storage medium, any of the Transformer-based 3D instance segmentation methods in the above embodiments can be stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above methods.

[0129] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A 3D instance segmentation method based on Transformer, characterized in that, include: Obtain multi-level features and seed points from the input point cloud; Features of aggregation seed points; The query point is obtained by filtering the aggregated seed points through translation operations and non-maximum suppression operations guided by intersection-union ratio; Multi-level features are divided into disjoint sets based on the similarity between multi-level features and the features of the query point; Cluster the query points and their corresponding sets to obtain the query output; The query output is input into a multilayer perceptron, which outputs the instance segmentation result. Among them, the seed points after aggregation are filtered by translation operation and non-maximum suppression operation guided by intersection-union ratio to obtain query points, including encoding the query points with the real center points to obtain interference-free query points; The true center point is obtained by resampling the center point of the instance from the true annotation, satisfying the following: in, N Indicates a Gaussian distribution. Indicates deviation, This represents the mean; The process involves clustering the query points and their corresponding sets to obtain the query output. This includes determining the query output features based on the decluttered query points, and includes the following steps: The query output is matched with the ground truth labels in the dataset. Queries with minimum matching cost are paired with perturbed ground truth centroids to form positive sample pairs. Positive sample queries are encouraged to have output features consistent with the output features of their corresponding centroids. Noisy background queries that do not match are removed from instances based on contrastive loss, which satisfies the following conditions: in, d(., .) It is a distance metric function. ε It refers to the temperature parameter in comparative learning; K The number of queries; The actual number of labeled items. β For accurate labeling The number of true centroids of the generated perturbations; query i and real labeling j The perturbation center points constitute positive sample pairs. Representative query i Features ， The first true label j represents the first true label. m Characteristics of each disturbance point; The step of dividing multi-level features into disjoint sets based on the similarity between multi-level features and the features of the query point includes the following steps: By using arg max operation The allocation is as follows: in It is a query and main characteristics dot product For multi-level features Level features.

2. The 3D instance segmentation method based on Transformer according to claim 1, characterized in that, The characteristics of the aggregation seed point include the following steps: Based on Mask3D, the original point cloud is divided into multiple superpoints. Then, the last level of the multi-level features is analyzed based on this superpoint division. Perform average pooling to obtain superpoint features ; Superpoint features and seed point features Perform cross-attention: in C is the number of feature channels. It is a linear mapping layer; After updating the seed point features using cross-attention, self-attention is used to update the seed point features: in , , ; After the self-attention update, a feedforward neural network composed of multilayer perceptrons is used to update the features of the seed points.

3. The 3D instance segmentation method based on Transformer according to claim 2, characterized in that, The process of filtering and aggregating seed points using translation and intersection-union ratio-guided nonmaximum suppression operations to obtain query points includes the following steps: Predict each seed point using a multilayer perceptron. Relative to its matching instance The coordinate offset of the center point; after obtaining this coordinate offset, add it to the original coordinates. The details are as follows: ,in It refers to characteristics; The seed point is obtained by calculating the dot product between the superpoint feature and the seed point feature. Corresponding instance mask The final mask is obtained by applying a threshold of 0.5 to the output of the Sigmoid function, as shown below: ； Using the mask obtained above Calculate the cross-union ratio (CUNR) and use CUNR-guided nonmaximum suppression to filter out redundant seed points.

4. The 3D instance segmentation method based on Transformer according to claim 1, characterized in that, The process of obtaining multi-level features and seed points from the input point cloud includes the following steps: Multi-level features are obtained by encoding the position and color information of the input point cloud using a 3D-UNet based on sparse convolution. Seed points obtained by sampling the input point cloud at the furthest distance. .

5. A system for performing the Transformer-based 3D instance segmentation method according to any one of claims 1-4, characterized in that, include: Feature aggregation module: Aggregates features of seed points through cross attention, self-attention, and feedforward neural networks; The seed points are obtained by sampling the input point cloud at the furthest distance, and the features of the seed points are multi-level features extracted from the input point cloud by 3D-UNet with sparse convolution. Set grouping module: Seed points are filtered using translation operations and non-maximum suppression operations guided by intersection-union ratio to obtain query points; Clustering layer module: Based on the similarity of multi-level features and the features of the de-interference query points, the multi-level features are divided into disjoint sets; the de-interference query points and their corresponding sets are processed through cross attention, attention and feedforward neural networks to obtain the query output; Denoising module: Matches the query output with the real labels in the dataset. Queries with minimum matching cost form positive sample pairs with the perturbed real centroids. It encourages the output features of positive sample queries to be consistent with the output features of the corresponding centroids, and uses contrastive loss to move mismatched noisy background queries away from instances, thus obtaining the output of denoised queries. Output module: The output of the denoised query is processed by a multilayer perceptron to output the instance segmentation result.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs a 3D instance segmentation method based on Transformer as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs a 3D instance segmentation method based on Transformer as described in any one of claims 1 to 4.

Images