An autoregressive generative point cloud converter pre-training method

Abstract

The present invention provides an autoregressive generative point cloud converter pre-training method. The double masking strategy is adopted to effectively reduce the redundancy of point cloud data and provide effective pre-training tasks, which has a significant effect on promoting beneficial representation learning and enhancing the generalization ability of pre-trained models. The relative direction hint can prevent the model from overfitting the order of image blocks, thereby improving the generalization ability of the model in downstream tasks. The extractor-generator structure, especially when combined with a deep generator, can reduce the gap between generation and downstream tasks, thereby achieving better performance overall. In terms of specific effects, this method has stronger generalization ability than existing methods, and compared with existing methods, this method is the first to explore the pre-training effect of single-modal large point cloud models and achieves the best level among all self-supervised methods.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional computer vision technology, and in particular relates to a pre-training method for an autoregressive generative point cloud converter. Background Art

[0002] Self-supervised learning has also been widely studied in point cloud representation learning. Among contrasting methods, DepthContrast utilizes enhanced depth maps to construct global features for instance discrimination. MVIF introduces cross-modal and cross-view invariance constraints to achieve self-supervised modality- and view-invariant feature learning. Recently, self-supervised generative methods, such as Point-BERT, Point-MAE, and Point-M2AE, have been proposed and show great potential. Point-MAE extends MAE by randomly masking point patches and reconstructing the masked regions. Point-M2AE also utilizes a layered transformer architecture to design a corresponding masking strategy. However, recent generative methods still suffer from the problem of global object shape leakage, which ultimately limits their ability to model global structure and generalize effectively. In addition to the aforementioned methods, a new approach has been proposed that applies cross-modal information and knowledge transfer from image or language models to 3D learning. Among them, ACT utilizes a cross-modal autoencoder as a teacher model to acquire implicit knowledge from other modalities. Unlike these methods, our work explores learning intrinsic properties of point clouds without relying on cross-modal information or a teacher model.

[0003] Inspired by the rapid development of contemporary self-supervised learning methods, researchers have extended self-supervised learning to point cloud tasks. In the field of point cloud self-supervised learning, occluded data modeling has become a mainstream approach due to its effectiveness in various point cloud processing tasks. However, occluded data modeling methods rely on location information to specify the reconstruction area, which in the point cloud field will reveal the shape of the overall object even under strict occlusion, ultimately hindering their ability to learn strong representations of point clouds. In addition to the above methods, a new method has recently been proposed in the field of point cloud self-supervision, which applies cross-modal information and knowledge transfer from image or language models to point cloud representation learning. However, this method relies on trained image and language models, making this type of method lack versatility and flexibility. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an autoregressive generative point cloud converter pre-training method to improve the generalization ability.

[0005] A pre-training method for an autoregressive generative point cloud converter is proposed. The pre-training network includes a point cloud sequence generator and a self-generated pre-trainer. The point cloud sequence generator includes a point cloud block segmentation module, a sorting module, and an embedding module. The self-generated pre-trainer includes an extractor, a generator, and a prediction head module. The extractor and generator are implemented using a converter decoding block.

[0006] Feed the given point cloud X into the point cloud sequence generator;

[0007] The point cloud block segmentation module samples the point cloud X using the farthest point sampling FPS to obtain multiple center points C, the number of which is represented by n; then uses the KNN algorithm to select the k nearest points of each center point from X to construct n point blocks P;

[0008] The sorting module uses Morton coding to embed the coordinates of the center point C into a one-dimensional space and sorts it to obtain the Morton curve; then, the point blocks P are arranged in the same order accordingly; and the sorted center point sequence C is obtained. s and point block sequence P s ;

[0009] The embedding module uses the PointNet network to embed the sorted point blocks P s Embedded into a D-dimensional token T;

[0010] The token T is fed into the extractor, where the self-attention mechanism adopts a double mask strategy. The self-attention with double mask is shown below:

[0011]

[0012] In the above formula, Q, K, V are the results of encoding T using D channels with different weights; when setting the double masking matrix M d When , the elements that need to be masked are set to 0, and the elements that do not need to be masked are set to 1;

[0013] Token T is processed in the extractor to obtain token

[0014]

[0015] In the above formula, APE represents the encoding of mapping the center point coordinates to the absolute position using sinusoidal position encoding PE;

[0016] The token to be extracted Input to the generator, generating point token T g :

[0017]

[0018] in, Representation token 1 to n' elements, n' = n-1; RDP represents the point block sequence P s Relative direction indication between midpoint blocks;

[0019] The prediction header module is used to transform the token T g Projected into a vector, after two layers of MLP processing, the number of output channels is equal to the total number of points in a point block, and then reconstructed into a prediction point block set P with a dimension of n'×k×3 pd : Among them, the prediction head module includes two layers of MLP, each layer consists of an FC layer and a ReLU activation function;

[0020] For the prediction point block set P pd and corresponds to the sorting point block P s The truth point block P of the last n' blocks gt , generating chamfer distances in the form of l1 and l2 to define the loss The loss of the entire network is

[0021] in,

[0022]

[0023] Then, based on the network loss Supervised training of the pre-trained network using a labeled mixed dataset to fuse semantics from multiple sources;

[0024] Finally, the pre-trained network is fine-tuned on the target dataset to complete the pre-training.

[0025] The present invention has the following beneficial effects:

[0026] The present invention provides an autoregressive generative point cloud converter pre-training method. The double masking strategy is adopted to effectively reduce the redundancy of point cloud data and provide effective pre-training tasks, which has a significant effect on promoting beneficial representation learning and enhancing the generalization ability of pre-trained models. The relative direction hint can prevent the model from overfitting the order of image blocks, thereby improving the generalization ability of the model in downstream tasks. The extractor-generator structure, especially when combined with a deep generator, can reduce the gap between generation and downstream tasks, thereby achieving better performance overall. In terms of specific effects, this method has stronger generalization ability than existing methods, and compared with existing methods, this method is the first to explore the pre-training effect of point cloud single-modal large models and has achieved the best level among all self-supervised methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1The overall architecture of the method of the present invention is as follows: (a) the input point cloud is divided into multiple point patches, which are then sorted and arranged in sequence; (b) a transformer-decoder-based extractor-generator and a double masking strategy are used to autoregressively predict point patches. DETAILED DESCRIPTION

[0028] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0029] The method proposed in this paper (PointGPT) consists of two stages: pre-training and fine-tuning. The overall process of this method is as follows Figure 1 As shown. Given a point cloud The point cloud sequence generator is used to construct an ordered sequence of point patches. This is achieved by dividing the point cloud into irregular patches and arranging them in Morton order. The generated sequence is then fed into the extractor to learn high-level latent representations, and the generator predicts subsequent point patches in an autoregressive manner. During the fine-tuning phase, the generator is disconnected and the high-level latent representations learned by the extractor are utilized for downstream tasks.

[0030] 1. Point Cloud Sequence Generator

[0031] Unlike the field of natural language processing (NLP), point clouds lack a predefined vocabulary. Point clouds are sparse, disordered structures. To obtain an ordered point cloud sequence where each component captures rich geometric information, the proposed point cloud sequence generator consists of three modules: a point cloud block segmentation module, a sorting module, and an embedding module.

[0032] Point cloud block segmentation module: Considering the inherent sparse and disordered nature of point clouds, this paper uses the Farthest Point Sampling (FPS) and K-Nearest Neighbor (KNN) algorithm to process the input point cloud to obtain center points and point blocks. Specifically, given a point cloud X (containing N points), this paper first uses FPS to sample n center points C. Then, the KNN algorithm is used to select the k nearest points to each center point in X to construct n point blocks P. The overall process is as follows:

[0033] C=FPS(X),

[0034] P=KNN(C,X),

[0035] Sorting module: To address the inherent disorder of point clouds, the present invention organizes the obtained point blocks into a consistent sequence according to their center points. Specifically, the center point coordinates are embedded into a one-dimensional space using Morton Code and sorted to obtain the Morton curve. Then, the point blocks are arranged in the same order accordingly. Specifically, the sorted center point sequence C s and point block sequence Ps The acquisition is as follows:

[0036] I=argmax(MortonCode(C)),

[0037] C s ,P s =C[I],P[I],

[0038] Among them, argmax represents the sorting operation from large to small.

[0039] Embedding module: In this invention, the PointNet network is used to extract rich geometric information from each point block, including a multi-layer perceptron and a maximum pooling layer. In order to facilitate training convergence, for each point, its normalized coordinate relative to the center point is used. Specifically, the sorted point block P s is embedded into the D-dimensional token T. Specifically, the sorted point cloud block P s is embedded into a token T of dimension D:

[0040] T=PointNet(P s ),

[0041] 2. Converter decoder with double mask strategy

[0042] A straightforward extension of GPT is to leverage a transformer decoder for autoregressive pre-training, then fine-tune all pre-trained parameters for downstream tasks. However, this approach is limited by the gap between generation and downstream tasks, as well as the lack of feature richness caused by the difference in information density between words and points, which limits the generalization ability of the pre-trained model. To address this, we introduce a generator specifically designed for generation tasks, construct an extractor-generator architecture to bridge the gap, and propose a dual masking strategy to encourage the learning of useful features.

[0043] Double Masking Strategy: To further promote the learning of useful features, a double masking strategy is proposed, which adds masking of a certain proportion of forward tokens in the transformer decoder so that these tokens are not paid attention to. Double Masking M d The schematic diagram is as follows Figure 1 (b) shown.

[0044] Extractor: The extractor of this invention is composed entirely of transformer decoding blocks, in which the self-attention mechanism adopts a double mask strategy. The self-attention with double mask is shown below:

[0045]

[0046] Among them, Q, K, V are the results of encoding T using D channels with different weights. d In , masked elements are set to 0 and unmasked elements are set to 1. Each point token depends only on itself and the unmasked preceding point token, thus obtaining the potential representation Considering that point clouds are represented in normalized coordinates and global structure is very important for point cloud understanding, a sinusoidal position encoding (PE) is used to map the center coordinates to an absolute position encoding (APE). Position encoding is added to each transformer block to provide position information and incorporate global structure information. The process in the extractor structure can be expressed as:

[0047]

[0048] Generator: The structure of the generator is similar to that of the feature extractor, but contains fewer transformer modules. As input, generate a point token T g , used for the next prediction head. To avoid the point sequence order being affected by center point sampling, which leads to unclear designation of subsequent point blocks and overfitting to the sorting curve, hindering the network from effectively learning meaningful features of the point cloud, the present invention provides a relative direction indication in the generator as a network hint while avoiding revealing the position of the mask block and the overall object shape of the point cloud. The formula for the relative direction hint RDP is as follows:

[0049] RDP i =PE((C i+1 -C i ) / ‖C i+1 -C i ‖2),i∈{1,...,n'},

[0050] Where n'=n-1.

[0051] The process in the generator structure can be expressed as:

[0052]

[0053] Representation token 1 to n' elements in it.

[0054] Prediction head: The prediction head is used to predict the subsequent set of points in the coordinate space. It consists of two layers of MLP, each with FC layer and ReLU activation function. The prediction head will token T gProjected into vectors, where the number of output channels is equal to the total number of coordinates in a point block. These vectors are then reshaped into the predicted point set P pd , to adapt to the output structure of the network:

[0055] P pd =Reshape(MLP(T g ),

[0056] 3. Generate Target

[0057] The goal of generating each point block is to predict the coordinates of the points in the next point block. Given the predicted point block P pd and corresponds to the sorting point block P s The truth point block P of the last n' blocks gt , generating loss The chamfer distances are defined using the form of l1 and l2, which are expressed as and Specifically, the generation loss is calculated as in n∈{1,2}, defined as:

[0058]

[0059] 4. Post-pre-training

[0060] Transfer learning schemes are a key step in applying pre-trained models to downstream tasks. Currently, current point cloud self-supervised methods are fine-tuned directly on the target dataset, which is still prone to overfitting due to limited semantic supervision signals. To alleviate this problem and facilitate the training of high-capacity models, the present invention adopts the intermediate fine-tuning strategy used in computer vision and proposes a post-pre-training stage for this method. In this stage, the present invention uses a labeled hybrid dataset that collects and aligns multiple existing annotated datasets. Supervised training is performed on this dataset to fuse semantics from multiple sources, and then fine-tuned on the target dataset to transfer general semantics to task-specific knowledge.

[0061] 5. Application

[0062] This method is used to perform semantic segmentation on point cloud data collected by the LiDAR equipped on autonomous vehicles. Specifically, the method first pre-trains the network using a large-scale public dataset, enabling the model to extract features from point cloud data. The resulting pre-trained network is then post-trained on a large-scale annotated dataset for point cloud semantic segmentation, enabling the pre-trained model to obtain multi-source supervision information from multiple datasets and enhancing the model's ability to extract semantic information. Finally, the post-pre-trained model is fine-tuned on a public dataset for autonomous point cloud semantic segmentation, transferring the multi-source semantic information to the semantic segmentation task. After completing the pre-training-post-training-fine-tuning network training process, the method deploys the fine-tuned model parameters on the autonomous vehicle, using the point cloud data collected by the LiDAR as input and passing it through the proposed network to output the point cloud semantic segmentation results. Under these application conditions, the extractor of this method is composed of a cascade of 12 transformer-decoder blocks with a dual masking mechanism, and the generator is composed of a cascade of 4 transformer-decoder blocks with a dual masking mechanism.

[0063] The present invention introduces Morton coding to sort point blocks and generate an ordered sequence of point clouds, so that the autoregressive generative pre-training paradigm can be extended to the field of point clouds; a double-mask strategy is proposed to reduce the redundancy of point clouds and provide a challenging generative pre-training task, so that pre-training can effectively learn the intrinsic features of point clouds; the extractor-generator framework is introduced into the converter decoder, so that the present method can effectively bridge the gap between the generative pre-training task and specific downstream tasks and improve the richness of the extracted features; a large number of experiments show that the method of the present invention outperforms the comparison method and achieves the best level in classification tasks on the ScanObjectNN and ModelNet40 datasets.

[0064] Example:

[0065] Table 1 shows the classification results on the ScanObjectNN and ModelNet40 datasets. Three settings were used on the ScanObjectNN dataset, with PB-T50-RS being the most challenging setting. In addition, the accuracy of 1k and 8k points is reported on the ModelNet40 dataset, and all results are expressed as percentages.

[0066] Table 1

[0067]

[0068] As shown in Table 2, few-shot object classification on ModelNet40. In each experimental setting, 10 independent trials are conducted and the average accuracy (%) with standard deviation is reported.

[0069] Table 2

[0070]

[0071] Methods using cross-modal information and teacher models

[0072]

[0073] Table 3 shows the part segmentation on the ShapeNetPart dataset. The mIoU of all categories (Cls.) and the mIoU of all instances (Inst.) are reported.

[0074] Table 3

[0075]

[0076] 1. Model settings

[0077] PointGPT was trained using the ViT-S configuration (zhai2022scaling), referred to as PointGPT-S. This is consistent with previous SSL methods for occluded point modeling, allowing for direct comparison with these previous approaches. Furthermore, we explore the performance of PointGPT at larger model capacities, extending it to the ViT-B and ViT-L configurations, referred to as PointGPT-B and PointGPT-L, respectively.

[0078] 2. Object classification tasks on real datasets

[0079] In order to evaluate the practical application performance of the proposed method on real-world datasets, the present invention applies the pre-trained model to the ScanObjectNN dataset, which contains approximately 15,000 objects extracted from real-world indoor scans. The present invention conducts experiments under three different settings: OBJ-BG, OBJ-ONLY, and PB-T50-RS. Table 1 shows the experimental results, and the method of the present invention achieves the most advanced performance level. In particular, the present invention's PointGPT-S has similar capacity and training data to Point-MAE, and achieves competitive performance among unimodal self-supervised learning methods. In addition, even when compared with Recon and ULIP using cross-modal information and teacher models, the large model of the present invention PointGPT-B achieves better performance, and the accuracy of PointGPT-L is at least 1.8% higher.

[0080] 3. Object classification task on clean object dataset

[0081] This study evaluated the pre-trained models on the ModelNet40 dataset, which contains 12,311 clean 3D CAD models covering 40 categories. For fair comparison, a standard voting method was used during testing, and the input point clouds of all compared methods only contained coordinate information without normal information. The experimental results are shown in Table 1. The method of the present invention achieved the best performance among all methods. Specifically, PointGPT-S of the present invention surpassed all other single-modal SSL methods. Even when compared with Recon and ULIP using cross-modal information and teacher models, the extended model PointGPT-L of the present invention was able to achieve better performance.

[0082] 4. Few-shot learning tasks

[0083] To demonstrate the generalization capability of our method, we conducted few-shot learning experiments on the ModelNet40 dataset. These experiments consisted of four experiments using an n-class, m-sample setup, where n∈{5,10} represents the number of randomly selected classes and m∈{10,20} represents the number of randomly sampled objects per selected class. For each setup, we conducted 10 independent trials, and reported the mean accuracy and standard deviation. The results are shown in Table 2. Our method achieved state-of-the-art performance in all settings, demonstrating its ability to learn new tasks using limited training data.

[0084] 5. Component segmentation task

[0085] To demonstrate the representation learning capabilities of our method, we evaluated it on the ShapeNetPart dataset, which includes 16,881 objects from 16 categories. The point cloud was sampled to 2,048 points. The experimental results are shown in Table 3. Our PointGPT-L performs best among all compared methods.

[0086] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pre-training method for an autoregressive generative point cloud converter, characterized in that Establish a pre-training network, including a point cloud sequence generator and a self-generated pre-trainer; wherein the point cloud sequence generator includes a point cloud block segmentation module, a sorting module, and an embedding module; the self-generated pre-trainer includes an extractor, a generator, and a prediction head module; the extractor and generator are implemented using a transformer decoding block; Feed the given point cloud X into the point cloud sequence generator; The point cloud block segmentation module samples the point cloud X using the farthest point sampling FPS to obtain multiple center points C, the number of which is represented by n; then uses the KNN algorithm to select the k nearest points of each center point from X to construct n point blocks P; The sorting module uses Morton coding to embed the coordinates of the center point C into a one-dimensional space and sorts it to obtain the Morton curve; then, the point blocks P are arranged in the same order accordingly; and the sorted center point sequence C is obtained. s and point block sequence P s ; The embedding module uses the PointNet network to embed the sorted point blocks P s Embedded into a D-dimensional token T; The token T is fed into the extractor, where the self-attention mechanism adopts a double mask strategy. The self-attention with double mask is shown below: In the above formula, Q, K, V are the results of encoding T using D channels with different weights; when setting the double masking matrix M d When , the elements that need to be masked are set to 0, and the elements that do not need to be masked are set to 1; Token T is processed in the extractor to obtain token In the above formula, APE represents the encoding of mapping the center point coordinates to the absolute position using sinusoidal position encoding PE; The token to be extracted Input to the generator, generating point token T g : in, Representation token 1 to n' elements, n' = n-1; RDP represents the point block sequence P s Relative direction indication between midpoint blocks; The prediction header module is used to transform the token T g Projected into a vector, after two layers of MLP processing, the number of output channels is equal to the total number of points in a point block, and then reconstructed into a prediction point block set P with a dimension of n'×k×3 pd : Among them, the prediction head module includes two layers of MLP, each layer consists of an FC layer and a ReLU activation function; For the prediction point block set P pd and corresponds to the sorting point block P s The truth point block P of the last n' blocks gt , generating chamfer distances in the form of l1 and l2 to define the loss The loss of the entire network is in, Then, based on the network loss Supervised training of the pre-trained network using a labeled mixed dataset to fuse semantics from multiple sources; Finally, the pre-trained network is fine-tuned on the target dataset to complete the pre-training.

Images