Point cloud model pre-training method based on mask auto-encoder and contrast learning

Abstract

The invention discloses a point cloud model pre-training method based on a mask auto-encoder and contrast learning, and the method comprises the following steps: generating a point cloud embedded mark block and an image embedded mark block based on the preprocessed point cloud and image data, and carrying out the mask alignment, and obtaining the point cloud mark block and the image mark block after the mask alignment; carrying out feature extraction on the basis of the point cloud mark blocks after mask alignment by adopting a point cloud Transform encoder, and carrying out feature extraction on the basis of the image mark blocks after mask alignment by adopting an image Transform encoder; based on the extracted point cloud features and image features, performing self-supervision pre-training through point cloud feature prediction and a point cloud mask reconstruction task to obtain a pre-trained point cloud Transform encoder; the pre-trained point cloud Transform encoder is subjected to fine adjustment in a downstream three-dimensional object classification task, and a fine-adjusted Transform encoder is obtained; and inputting the test set data into the fine-tuned point cloud Transform encoder to realize category classification of the target object in the point cloud.

Description

Technical Field

[0001] The present invention belongs to the field of fully automated products and relates to a method for pre-training a point cloud model based on a masked autoencoder and contrastive learning. Background Art

[0002] In the field of 3D computer vision, the pre-training method of point cloud models has become one of the research focuses. As an important 3D data representation form, point clouds have wide applications in the fields of intelligent manufacturing, autonomous driving, augmented reality, etc. Previous supervised pre-training methods rely on a large amount of labeled point cloud data for training. However, in the case of scarce data or difficult annotation, the effect of supervised pre-training is significantly affected. In recent years, researchers have begun to turn their attention to the pre-training method of point cloud models based on self-supervised learning, aiming to learn robust and general feature representations from unlabeled data to alleviate the time-consuming and laborious data annotation problem. Through self-supervised learning, the model can obtain a more comprehensive and generalizable representation ability without data annotation, overcoming the limitations of supervised methods in the case of scarce data or difficult annotation.

[0003] Most of the existing point cloud self-supervised pre-training methods focus on single point cloud data, which leads to overly single feature information extracted, and thus insufficient generalization ability in downstream tasks. Although there have been related studies on multi-modal self-supervised pre-training methods, the existing methods fail to fully utilize the correlation between two different modal data and do not fully consider the complementarity between different modal data, which may lead to the loss of important information in the feature correspondence process, thus affecting the performance in downstream tasks. Therefore, the research on how to effectively transfer self-supervised pre-training methods to downstream tasks remains a major challenge. Summary of the Invention

[0004] In order to solve the above problems, the technical solution adopted by the present invention is: a method for pre-training a point cloud model based on a masked autoencoder and contrastive learning, including the following steps:

[0005] Construct a pre-training dataset to obtain point cloud and rendered image data;

[0006] Obtain the point cloud and image data in the pre-training dataset, and preprocess the point cloud and image data;

[0007] Construct a point cloud pre-training model based on a masked autoencoder and contrastive learning. The process of pre-training the point cloud pre-training model is as follows:

[0008] Based on the preprocessed point cloud and image data, generate point cloud embedding token blocks and image embedding token blocks and perform mask alignment to obtain the masked-aligned point cloud token blocks and image token blocks;

[0009] For the point cloud labeled blocks after mask alignment, a point cloud Transformer encoder is used for feature extraction, and for the image labeled blocks, an image Transformer encoder is used for feature extraction;

[0010] Based on the extracted point cloud features and image features, self-supervised pre-training is carried out through point cloud feature prediction and point cloud mask reconstruction tasks to obtain a pre-trained point cloud Transformer encoder;

[0011] The pre-trained point cloud Transformer encoder is fine-tuned in the downstream 3D object classification task to obtain a fine-tuned Transformer encoder;

[0012] The test set data is input into the fine-tuned point cloud Transformer encoder to classify the types of target objects in the point cloud.

[0013] Furthermore: The process of constructing the pre-training dataset to obtain point cloud and rendered image data is as follows:

[0014] Render the CAD models in the V2 version of ShapeNet, scan the models from different perspectives around the CAD models through a virtual camera, and then generate depth images and RGB-D images corresponding to the perspectives;

[0015] Restore the point cloud corresponding to the perspective from the depth map according to the rendering parameters;

[0016] After rendering, randomly select samples from this dataset to form a pre-training dataset, which includes point clouds and images corresponding to the perspectives, and the external parameter matrix and internal parameter matrix parameters during the rendering process.

[0017] Furthermore: The process of generating point cloud embedding labeled blocks and image embedding labeled blocks is as follows:

[0018] First, divide the point cloud data into blocks. The block division method is the same as that of the mask autoencoder for point cloud self-supervised learning. For the input point cloud, sample G central points from the point cloud using the farthest sampling algorithm, and then based on each central point, use the K-nearest neighbor algorithm to select N neighboring points for it;

[0019] For each point cloud block \(P\), use a mini-PointNet model to perform embedding to obtain point cloud embedding labeled blocks. For the image part, it is the same as the VIT model. After performing convolution operations using two-dimensional convolution and flattening the two dimensions of \(H\) and \(W\), image embedding labeled blocks are obtained.

[0020] Furthermore: The process of generating point cloud embedding tokens and image embedding tokens based on the preprocessed point cloud and image data and performing mask alignment to obtain the masked-aligned point cloud tokens and image tokens is as follows:

[0021] For the point cloud embedding tokens, perform random masking to obtain visible point cloud tokens and masked tokens;

[0022] For the image embedding tokens, process them in the following way using alignment masks:

[0023] Obtain the pixel coordinates of the center point of the masked point cloud tokens projected onto the image through the extrinsic matrix and the intrinsic matrix;

[0024] Mask the image embedding tokens where the pixel coordinates are located to obtain visible image tokens corresponding to the visible point cloud tokens and image masked tokens corresponding to the masked tokens.

[0025] Furthermore: The process of self-supervised pre-training based on the extracted point cloud features and image features through point cloud feature prediction and point cloud mask reconstruction tasks is as follows:

[0026] During the point cloud feature prediction process, pass the point cloud features through the decoder and the prediction head to obtain the prediction vector, and pass the image features through the projection head to obtain the projection vector;

[0027] Maximize the similarity between the prediction vector and the projection vector to calculate the cross-modal contrast learning loss.

[0028] During the mask reconstruction process, jointly input the image features and the point cloud features into the decoder for fusion, and then pass the fused features through the point cloud prediction head to obtain the point cloud prediction value;

[0029] Calculate the loss between the point cloud prediction value and the ground truth.

[0030] A method for pre-training a point cloud model based on a masked autoencoder and contrastive learning provided by the present invention uses a masked autoencoder based on the PointMAE model, which is divided into two parts: a Transformer encoder feature extraction module and a Transformer decoder mask reconstruction module. It can extract point cloud features and reconstruct the masked point cloud blocks. It has the following advantages:

[0031] A novel method for pre-training a point cloud model is proposed. This method uses a masked autoencoder to pre-train the point cloud model. This method is not limited to the traditional pre-training tasks of point cloud models in a single modality, but instead introduces the image modality to assist in pre-training the point cloud model;

[0032] A multi-modal mask alignment strategy is proposed, aiming to align mask blocks in different modalities, thereby enhancing the model's perception ability of multi-modal information. This strategy not only enhances the model's 3D understanding ability but also provides a robust representation for downstream tasks.

[0033] Two self-supervised pre-training tasks are designed. The model learns through a feature prediction task and a point cloud mask reconstruction task to help the model fully explore the potential correlation information between different modalities, thereby enhancing the model's generalization ability. Brief Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0035] Figure 1 It is the overall flowchart of the method of the present invention;

[0036] Figure 2 It is the training process of the point cloud model of the present invention;

[0037] Figure 3 It is the rendered point cloud and image effect diagram; where (a) is view 1, (b) is view 2, (c) is view 3;

[0038] Figure 4 It is the point cloud projected onto the image effect diagram;

[0039] Figure 5 It is the detailed process diagram of generating point cloud and image embedding blocks and performing mask alignment;

[0040] Figure 6 It is the detailed structural diagram of the point cloud feature prediction and mask reconstruction tasks;

[0041] Figure 7 It is the point cloud reconstruction effect diagram on the ShapeNet test dataset. (a) is the ground truth, (b) is the case of 60% mask, (c) is the predicted value. Detailed Embodiments

[0042] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to detail the present invention.

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. The following description of at least one exemplary embodiment is actually only illustrative and in no way limits the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0044] Figure 1 It is the overall flowchart of the method of the present invention;

[0045] Figure 2 It is the training process of the point cloud model of the present invention;

[0046] A method for pre-training a point cloud model based on a masked autoencoder and contrastive learning includes the following steps:

[0047] S1: Construct a pre-training dataset to obtain point clouds and rendered image data;

[0048] S2: Obtain the point clouds and image data in the pre-training dataset, and preprocess the point clouds and image data;

[0049] S3: Construct a point cloud pre-training model based on a masked autoencoder and contrastive learning. The process of pre-training the point cloud pre-training model is as follows:

[0050] S4: Based on the preprocessed point clouds and image data, generate point cloud embedding token blocks and image embedding token blocks and perform masked alignment to obtain the masked-aligned point cloud token blocks and image token blocks;

[0051] S5: Extract features using a point cloud Transformer encoder based on the masked-aligned point cloud token blocks, and extract features using an image Transformer encoder based on the masked-aligned image token blocks;

[0052] S6: Based on the extracted point cloud features and image features, perform self-supervised pre-training through point cloud feature prediction and point cloud mask reconstruction tasks to obtain a pre-trained point cloud Transformer encoder;

[0053] S7: Fine-tune the pre-trained point cloud Transformer encoder in a downstream 3D object classification task to obtain a fine-tuned Transformer encoder;

[0054] S8: Input the test set data into the fine-tuned point cloud Transformer encoder to classify the target objects in the point cloud.

[0055] Steps S1 / S2 / S3 / S4 / S5 / S6 / S7 / S8 are executed in sequence;

[0056] The process of constructing a pre-training data set to obtain point cloud and rendered image data is as follows:

[0057] The constructed pre-training dataset is obtained based on the V2 version of ShapeNet;

[0058] The CAD model in the V2 version of ShapeNet is rendered using Blender software. The virtual camera simulated by Blender scans the model from different perspectives around the CAD model, and then generates depth images and RGB-D images at the corresponding perspectives.

[0059] According to the rendering parameters, the depth map is restored to the point cloud at the corresponding perspective;

[0060] After rendering, samples are randomly selected from the dataset to form a pre-training dataset, which contains point clouds and images from corresponding perspectives, as well as the external and internal parameter matrices during the rendering process.

[0061] Further: the process of generating the point cloud embedding marker block and the image embedding marker block is as follows:

[0062] First, the point cloud data is divided into blocks. The block division method used is consistent with the PointMAE model. For the input point cloud, the farthest sampling algorithm is used to sample G center points from the point cloud. Then, based on each center point, the K nearest neighbor algorithm is used to select N neighboring points for it.

[0063] PointMAE is a method whose full name is "Masked Autoencoders for Point Cloud Self-supervised Learning".

[0064] For each point cloud patch block PP, a mini-PointNet model is used to embed the point cloud embedding marker block. The image part is consistent with the VIT model. After a two-dimensional convolution operation, the H and W dimensions are flattened to obtain the image embedding marker block.

[0065] PointNet is fully known as: "Deep Learning on Point Sets for 3D Classification and Segmentation."

[0066] Furthermore, the process of obtaining the masked-aligned point cloud and image token blocks by aligning the generated point cloud embedding token blocks with the image embedding token blocks is as follows:

[0067] S31: For the point cloud embedding token blocks, perform random masking to obtain the visible point cloud token blocks and the masked token blocks;

[0068] S32: For the image embedding token blocks, process them in the way of aligned masking, specifically as follows:

[0069] S321: Obtain the pixel coordinates of the center point of the masked token blocks of the point cloud in the image through the extrinsic matrix and the intrinsic matrix;

[0070] S322: Mask the image embedding token blocks according to the pixel coordinates to obtain the visible image token blocks corresponding to the visible point cloud token blocks and the masked image token blocks corresponding to the masked tokens.

[0071] Furthermore, the process of feature extraction using the Transformer encoder in the model based on the masked-aligned point cloud and image token blocks is as follows: Feature extraction of the point cloud and the image is performed by passing the point cloud embedding block and the image embedding block through the same-structured Transformer encoders respectively.

[0072] Furthermore, the process of self-supervised pre-training based on the extracted point cloud and image features through point cloud feature prediction and point cloud mask reconstruction tasks is as follows:

[0073] S511: Pass the point cloud features through the decoder and the prediction head to obtain the prediction vector;

[0074] S512: Pass the image features through the projection head to obtain the projection vector;

[0075] S513: Maximize the similarity between the prediction vector and the projection vector to calculate the cross-modal contrastive learning loss.

[0076] S521: In the mask reconstruction process, jointly input the image features and the point cloud features into the decoder for fusion, and then pass the fused features through the point cloud prediction head to obtain the point cloud prediction value.

[0077] S522: Calculate the loss between the point cloud prediction value and the ground truth.

[0078] This embodiment is built with 1 GTX 3090 graphics card server as the test running environment for the method of the present invention. All experiments are carried out on the server for model training and testing. The specific hardware configuration information is shown in Table 1.

[0079] Table 1 Server Hardware Configuration

[0080] Configuration Specification CPU Intel i7-10700K CPU@3.8GHz GPU GTX3090 Memory 32GB Hard Disk 500GB + 6TB System Ubuntu18.04

[0081] This embodiment uses Pycharm as the development environment for the method of the present invention and Python as the programming language to complete the method design and development. The software environment included in this embodiment for the method to run includes: operating system Ubuntu 18.04, Python 3.8. The specific software environment is shown in Table 2.

[0082] Table 2 Software Environment

[0083] Software Environment Version Operating System Ubuntu 18.04 Development Environment Pycharm Language Python 3.8 Framework Pytorch 1.10.1

[0084] The process of constructing the pre-training dataset to obtain the point cloud and the rendered image data is as follows:

[0085] The constructed pre-training dataset is obtained based on the V2 version of ShapeNet;

[0086] Use Blender software to render the CAD models therein. The virtual camera simulated by Blender scans the models from different viewpoints around the CAD models, and then generates depth images and RGB-D images corresponding to the viewpoints;

[0087] Finally, restore the point cloud from the depth map according to the rendering parameters. The images rendered from different viewpoints and the point clouds corresponding to the viewpoints are as Figure 3 shown; where (a) Viewpoint 1, (b) Viewpoint 2, (c) Viewpoint 3; as can be seen from the figure, the lower part is the point cloud samples from different viewpoints, and the upper part are the images rendered from these viewpoints.

[0088] After rendering, randomly select 41,928 samples from this dataset to form the pre-training dataset; it includes the point cloud and the images corresponding to the viewpoints, as well as the external parameter matrix and internal parameter matrix parameters during the rendering process.

[0089] The process of obtaining the point cloud and image data in the pre-training dataset and preprocessing the point cloud and image data is as follows:

[0090] During pre-training, input the point cloud and image data in the pre-training dataset as the point cloud and the images corresponding to the viewpoints;

[0091] The selected point cloud sample is sampled into 2048 points using the farthest point sampling algorithm (FPS), and is divided into 64 point cloud blocks, each point cloud block contains 32 points. At the same time, the image corresponding to the viewing angle of the point cloud sample is selected as the input, and its size is adjusted to 224×224; data augmentation only normalizes the point cloud and the image.

[0092] Figure 4 It is the effect diagram of projecting the point cloud onto the image. Two methods are used to project the point cloud onto the image in the figure. The upper right image shows the result of projecting all points of the point cloud onto the image, and the lower right image shows the result of projecting the center points after dividing the point cloud into blocks onto the image.

[0093] The process of generating point cloud and image embedding blocks and performing mask alignment based on the preprocessed point cloud and image data to obtain the masked-aligned point cloud and image token blocks is as follows:

[0094] The detailed process of generating point cloud and image embedding blocks and performing mask alignment is as Figure 5 shown. Since the point cloud consists of disordered points in 3D space, it is different from the image that can be naturally divided into regular blocks;

[0095] Therefore, it is necessary to first divide the point cloud into blocks. The block division method is the same as that of PointMAE. For the input point cloud P i , the farthest sampling algorithm (FPS) is used to sample G center points from P i ;

[0096] Subsequently, based on each center point, the K-nearest neighbor algorithm (KNN) is used to select N neighboring points for it. The specific process can be described as follows:

[0097] G = FPS(P i ) (1)

[0098] PP = KNN(P i , G) (2)

[0099] For each point cloud Patch block PP, a mini-PointNet model is used to perform embedding to obtain the point cloud embedding token block T pc . For the image part, it is the same as VIT. After performing convolution operations using two-dimensional convolution and flattening the H and W dimensions, the image embedding token block T img is obtained; it can be described by the formula as follows:

[0100] T pc = PointNet(PP) (3)

[0101] T img = Reshape(Conv2D(img)) (4)

[0102] Finally, we mask the obtained point cloud embedding token blocks and image embedding token blocks.

[0103] For the point cloud embedding token blocks, we perform random masking to obtain the visible point cloud token blocks and the masked tokens

[0104] For the image embedding token blocks, we process them in the way of alignment masking.

[0105] First, we project the center point of the masked token block through the extrinsic matrix R t and the intrinsic matrix K to obtain its pixel coordinates u, v in the image. The effect diagram of the point cloud projected onto the image is as Figure 2 shown. The specific formula can be described as follows:

[0106]

[0107] Subsequently, we mask the image embedding token block where the pixel coordinates u, v are located to obtain the visible point cloud token blocks corresponding visible image token blocks the masked token blocks corresponding image mask token blocks

[0108] The process of feature extraction using the Transformer encoder in the point cloud model for the point cloud token blocks and image token blocks after mask alignment is as follows:

[0109] The structures of the point cloud encoder and the image encoder are the same, both consisting of standard Transformer blocks. During training, we only send the visible point cloud token blocks into the encoder Encoder pc to obtain the encoded token blocks

[0110] For the image, it is different. We send both the image embedding token block T img and the image mask token block into the encoder Encoder img and then obtain the encoded token blocks Specifically, it can be described as follows:

[0111]

[0112] The process of self-supervised pre-training to obtain a pre-trained point cloud model through point cloud feature prediction and point cloud mask reconstruction tasks based on the extracted point cloud features and image features is as follows:

[0113] The basic processes of the point cloud feature prediction and point cloud mask reconstruction tasks are as follows Figure 6 shown. As shown in the figure, in the feature prediction task, we perform contrastive learning by comparing the predicted point cloud features with the input features of the image branch, and calculate the contrastive loss. In the mask reconstruction task, we calculate the chamfer distance loss between the reconstructed point cloud patches and the ground truth.

[0114] Among them, the decoder of the feature prediction task has a structure similar to that of the encoder, but it has fewer Transformer blocks. The feature prediction head Pre f has the same structure as the image projection head Pro img , both of which are composed of two fully connected layers, and we only use batch normalization (BatchNorm) and non-linear activation (ReLU) after the first layer. We use the encoded visible point cloud token blocks and the learnable point cloud mask token blocks as inputs. After passing through the decoder and the feature prediction head Pre f , the predicted feature vector is denoted as F pre .

[0115] The image feature vector obtained by passing the encoded image token blocks through the projection head Pro img is denoted as F img . Specifically, it can be described as follows:

[0116]

[0117] The goal of the feature prediction task is to make the predicted point cloud features as similar as possible to the image features. For the features and , we use the contrastive learning loss to calculate, and the formula is as follows:

[0118]

[0119] where: i represents each sample, exp is the exponential function, N is the batch size, τ is the temperature coefficient, and sim(·) represents the cosine similarity.

[0120] Maximize the similarity between the predicted vector and the projected vector, and calculate the cross-modal contrastive learning loss. The batch loss function can be described as:

[0121]

[0122] The decoder in the point cloud mask reconstruction task is composed of cross-attention and self-attention layers. The point cloud prediction head Pre pcIt consists of a simple fully connected layer. We mark the encoded visible point cloud blocks Learnable point cloud mask token blocks And image visible token blocks As the input, the obtained decoded token block is denoted as D pre .

[0123] Subsequently, we project it into a vector through the point cloud prediction head Pre pc Finally, through a reshaping operation, the predicted point cloud block P is obtained pre . Specifically, it can be described as follows:

[0124]

[0125] P pre = Reshape(Pre pc (D pre )) (13)

[0126] The goal of the point cloud mask reconstruction task is to restore the coordinates of the points in each masked point cloud block. For the given predicted point cloud block P pre And the ground truth point cloud block P gt , we use the L2 chamfer distance to calculate the reconstruction loss:

[0127]

[0128] Where: x represents the predicted value, y represents the ground truth value, and ∥·∥2 represents the L2 norm. Finally, our overall self-supervised pre-training objective can be expressed as:

[0129] Loss total = Loss pc + λLoss f (15)

[0130] Where: Loss pc Is the reconstruction loss in formula (14), Loss f Is the contrastive learning loss in formula (11), and λ is the hyperparameter used to balance these two losses.

[0131] Figure 7 Is the point cloud reconstruction effect diagram of the pre-trained model tested on the ShapeNet test set. (a) is the ground truth case, (b) is the 60% masked case, and (c) is the predicted value. We mask 60% of the input data, and the test prediction results are shown as follows.

[0132] The process of fine-tuning the pre-trained point cloud Transformer encoder in the downstream 3D object classification task to obtain the fine-tuned Transformer encoder is as follows:

[0133] Since the pre-training dataset has changed, for the convenience of comparison, we reproduce the typical pre-training methods PointBert, PointMAE, and InterMAE using the same experimental settings. Subsequently, we transfer the pre-trained point cloud Transformer encoders of these methods to the downstream 3D shape classification task of point clouds for experiments. The experimental dataset uses the real-world dataset ScanObjectNN. ScanObjectNN is a more challenging real-world dataset with a different data distribution from the pre-training dataset. It consists of 2,902 point cloud samples in 15 categories from real cluttered background scans. This can better reflect the generalization performance of the pre-trained point cloud model. We conduct experiments on three split versions of ScanObjectNN: OBJ-ONLY, OBG-BG, and PB-T50-RS. The evaluation metric is the overall accuracy (OA) on all test samples. The fine-tuning classification experiment results on the ScanObjectNN dataset are shown in Table 3.

[0134] Table 3 Fine-tuning Classification Results on ScanObjectNN Dataset

[0135]

[0136] As can be seen from Table 3, our method improves by 8.61%, 8.09%, and 7.22% respectively compared to the Transformer baseline on the three different split versions of the dataset. On the most difficult PB-T50-RS split version, the accuracy of our method is 84.46%, which is 0.42% and 1.15% higher than the pre-trained Point-MAE and Inter-MAE respectively. The above analysis demonstrates the generalization of the point cloud model pre-trained using our method. Although we pre-train on the clean synthetic ShapeNet dataset, our method can also generalize well on real-world data, showing strong generalization ability.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for pre-training a point cloud model based on masked autoencoders and contrastive learning, characterized in that: Including the following steps: Construct a pre-training dataset to obtain point cloud and rendered image data; Obtain the point cloud and image data in the pre-training dataset, and preprocess the point cloud and image data; Construct a point cloud pre-training model based on masked autoencoder and contrastive learning. The process of pre-training the point cloud pre-training model is as follows: Based on the preprocessed point cloud and image data, generate point cloud embedding token blocks and image embedding token blocks and perform mask alignment to obtain the masked-aligned point cloud token blocks and image token blocks; Use a point cloud Transformer encoder to extract features based on the masked-aligned point cloud token blocks, and use an image Transformer encoder to extract features for the image token blocks; Based on the extracted point cloud features and image features, perform self-supervised pre-training through point cloud feature prediction and point cloud mask reconstruction tasks to obtain a pre-trained point cloud Transformer encoder; Fine-tune the pre-trained point cloud Transformer encoder in the downstream 3D object classification task to obtain a fine-tuned Transformer encoder; Input the test set data into the fine-tuned point cloud Transformer encoder to classify the types of target objects in the point cloud.

2. The pre-training method of the point cloud model based on the masked auto-encoder and contrastive learning according to claim 1, characterized in that: The process of constructing the pre-training dataset to obtain point cloud and rendered image data is as follows: Render the CAD models in the V2 version of ShapeNet, scan the models from different perspectives around the CAD models through a virtual camera, and then generate depth images and RGB-D images corresponding to the perspectives; Restore the point cloud corresponding to the perspective from the depth map according to the rendering parameters; After the rendering is completed, randomly select samples from this dataset to form a pre-training dataset, which includes the point cloud and the image corresponding to the perspective, as well as the external parameter matrix and internal parameter matrix parameters during the rendering process.

3. A pre-training method for a point cloud model based on a masked autoencoder and contrastive learning according to claim 1, characterized in that: The process of generating point cloud embedding token blocks and image embedding token blocks is as follows: First, divide the point cloud data into blocks. The division method is the same as that of the PointMAE model. For the input point cloud, use the farthest point sampling algorithm to sample G center points from the point cloud, and then based on each center point, use the K-nearest neighbor algorithm to select N neighboring points for it; For each point cloud block \(P\), use a mini-PointNet model to perform embedding to obtain point cloud embedding token blocks. For the image part, it is consistent with the VIT model. After performing convolution operations using two-dimensional convolution, flatten the two dimensions of \(H\) and \(W\) to obtain image embedding token blocks.

4. A method for pre-training a point cloud model based on a masked autoencoder and contrastive learning according to claim 1, characterized in that: The process of generating point cloud embedding token blocks and image embedding token blocks based on the preprocessed point cloud and image data, performing mask alignment, and obtaining the masked-aligned point cloud token blocks and image token blocks is as follows: For the point cloud embedding token blocks, perform random mask processing to obtain point cloud visible token blocks and masked token blocks; For the image embedding token blocks, perform processing in the way of alignment mask, specifically as follows: Obtain the pixel coordinates projected in the image of the center points of the masked token blocks of the point cloud through the external parameter matrix and internal parameter matrix; Mask the embedded marker blocks in the image according to the pixel coordinates to obtain the image visible marker blocks corresponding to the point cloud visible marker blocks and the image mask marker blocks corresponding to the masked markers.

5. A pre-training method for a point cloud model based on a masked autoencoder and contrastive learning according to claim 1, characterized in that: The process of self-supervised pre-training based on the extracted point cloud features and image features through point cloud feature prediction and point cloud mask reconstruction tasks is as follows: In the process of point cloud feature prediction, pass the point cloud features through the decoder and the prediction head to obtain the prediction vector, and pass the image features through the projection head to obtain the projection vector; Maximize the similarity between the prediction vector and the projection vector to calculate the cross-modal contrast learning loss. In the process of mask reconstruction, send the image features and the point cloud features into the decoder for fusion together, and then pass the fused features through the point cloud prediction head to obtain the point cloud prediction value; Calculate the loss between the point cloud prediction value and the ground truth.

Images