A CLIP-based three-dimensional point cloud few-shot classification method and system
By training the optimal projection angle globally in 3D point cloud classification and utilizing the CLIP visual encoder and learnable adapter, the problems of increased computation and decreased accuracy caused by fixed angles in the projected view are solved, achieving more efficient few-sample point cloud classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2024-04-22
- Publication Date
- 2026-07-24
AI Technical Summary
In existing technologies for few-sample classification of 3D point clouds, the fixed angle of the projected view leads to increased computation and decreased classification accuracy, especially since the optimal projection angle has not been found due to the differences and symmetry of objects.
The optimal projection angle is selected through overall training, and the point cloud is projected onto a two-dimensional image. The features of the multi-view image are encoded using a pre-trained CLIP visual encoder, and text prompts are generated to build a classifier. The feature fusion and classification are then performed in conjunction with a learnable inter-view adapter.
It achieves more accurate point cloud classification with fewer samples, reduces the dependence on a large amount of 3D labeled data, improves classification accuracy and training efficiency, and has strong versatility.
Smart Images
Figure CN118397341B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, and specifically relates to a method and system for few-shot classification of 3D point clouds based on CLIP. Background Technology
[0002] Three-dimensional point cloud shape classification aims to identify the category of each point cloud data. Point cloud shape classification has received more and more attention in recent years because it has a wide range of applications in many fields, such as computer vision, autonomous driving and robotics. In recent years, driven by large-scale language models, there is a popular method in the field of two-dimensional images. First, a large number of image text pairs are used for pre-training, and then downstream classification tasks are carried out. It has achieved good results in few-shot and zero-shot classification. Since the amount of point cloud data is small and the acquisition cost is high, few-shot and zero-shot point cloud classification is to train a model that can accurately classify new and unseen samples by using as few labeled samples as possible. It has certain significance in the field of point cloud classification. Various large-scale two-dimensional datasets provide rich samples for the pre-training model for high-quality and robust two-dimensional feature extraction. In contrast, it is difficult to acquire a large amount of point cloud data. The widely used 3D datasets are relatively small and have a limited number of categories, such as ModelNet40
[57] , with 9843 samples and 40 classes. ImageNet
[28] has 1 million samples and 1000 classes. Therefore, it is very difficult to obtain a well-pre-trained three-dimensional network for transfer learning. One current solution to this problem is to project the point cloud onto a 2D image at a fixed perspective angle to generate several fixed-direction multi-views. Then, the pre-trained CLIP model is classified using image text with few or zero samples. However, due to the differences and symmetries of objects, the above method may result in repeated projected images and failure to find the optimal projection angle for the object, which increases the computational load of the network and reduces the final classification accuracy.
[0003] In point cloud classification with few or zero samples, since the projected view directly affects the classification effect, how to improve the projection efficiency and obtain the optimal projected view is a technical problem that this method urgently needs to solve. Summary of the Invention
[0004] The present invention provides a method and system for few-shot classification of 3D point clouds based on CLIP. By training the whole system to select the optimal projection angle, the point cloud is projected onto a 2D image. The pre-trained CLIP visual encoder is used to encode the features of the multi-view image. Then, text prompts are generated based on the point cloud category and a classifier is constructed, thus realizing few-shot classification of point clouds.
[0005] This invention provides the following technical solutions:
[0006] On the one hand, a CLIP-based few-shot classification method for 3D point clouds includes:
[0007] Step 1: Use the PointNet point cloud classification network to extract features from the point cloud data, and obtain the point cloud angle category tensor based on the extracted features;
[0008] Step 2: Convert the point cloud angle category tensor obtained in Step 1 into Euler angles, and then convert the Euler angles into rotation matrices;
[0009] Step 3: Rotate the corresponding point cloud using the obtained rotation matrix, perform perspective projection on the rotated point cloud using the perspective projection module to obtain multi-view images, and extract multi-view image features using the ResNet network.
[0010] Objects present different characteristics when viewed from different angles. By performing perspective projection, we can obtain an angular view that best expresses the characteristics of the object, thereby obtaining the view image features that best reflect the characteristics of the object and achieving the most accurate classification.
[0011] The aforementioned perspective projection module is used to project the rotated point cloud onto a series of predefined image planes to generate a scattering depth map, thus preserving the original attributes of the point cloud from multiple views.
[0012] Projection process: Let (x, y, z) be the coordinates of a point p in the point cloud relative to the camera. Perspective projection is applied to obtain the two-dimensional coordinates of p at depth z (x1 = x / z, y1 = y / z). Multiple points can be projected onto the same discrete position on the image plane. To generate depth values at the image positions, a weighted average depth is calculated, with greater weight given to points closer to the view. This results in a multi-view image after perspective projection.
[0013] Step 4: Set a text preset template based on the known category point cloud to generate a known point cloud text prompt; input the known category point cloud text prompt and the multi-view image features obtained in Step 3 into the pre-trained CLIP, and output the predicted category of the point cloud;
[0014] The pre-trained CLIP obtains text features of point clouds with known categories and multi-view features of point clouds to be classified. The text features and multi-view features are compared for similarity, and the category corresponding to the text feature with the highest similarity is used as the text category of the multi-view image features to predict the point cloud category.
[0015] Before classification, the point cloud data sample set and the corresponding point cloud category labels are processed according to steps 1 to 4, and the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module and ResNet network are trained iteratively as a whole.
[0016] Furthermore, the process of obtaining the rotation matrix is as follows:
[0017] Step 2-1: Convert the point cloud angle category tensor to Euler angles;
[0018] Divide 360° into 36 angle intervals, which are 36 category tensors. The value of the category tensor ranges from 0 to 35. The center value of the angle interval in which the point cloud angle category tensor is located is taken as the center value of the interval in which the corresponding Euler angle value of the point cloud angle category tensor is located. The calculation formula is as follows:
[0019]
[0020] in, It is the Euler angle of the point cloud around the XYZ directions, N x N y N z This is to obtain the point cloud angle category tensor in the x, y, z directions using the features extracted from the point cloud by the PointNet point cloud classification network;
[0021] For example, if the class tensor is 0, the corresponding Euler angle is 5°; if the class tensor is 1, the corresponding Euler angle is 15°, and so on.
[0022] Step 2-2: Construct the point cloud rotation matrix based on the Euler angles of the point cloud;
[0023]
[0024]
[0025]
[0026]
[0027] Where Rotmat is the rotation matrix of the constructed point cloud. R y (β), R z (γ) is the rotation matrix of the point cloud around the X, Y, and Z directions.
[0028] Furthermore, the process of inputting the known category point cloud text prompts and the multi-view image features obtained in step 3 into the pre-trained CLIP to obtain the final multi-view encoded features of the point cloud to be classified is as follows:
[0029] Step 4-1: Encode the multi-view features of the point cloud to be classified using the visual encoder in the pre-trained CLIP to obtain the initial encoded features f. v f v ∈R 1×C ;
[0030] C represents the feature dimension, and R represents the real number space;
[0031] Step 4-2: Using a learnable inter-view adapter, the features encoded in Step 4-1 are concatenated to obtain compact global features of the point cloud;
[0032] Specifically, given M CLIP-encoded multi-view features of a point cloud, concatenate them into a feature Concate(f). 1-M ), Concate(f 1-M )∈R 1×MC And obtain compact global features of the point cloud through the first two layers of the learnable inter-view adapter;
[0033]
[0034] Among them, f global f represents the compact global features of a point cloud. global ∈R 1×C Concate(f) 1-M ) indicates that it will be regarded as Figure 1 Features are obtained by concatenating features to M, where M is the number of views, Concate means feature concatenation operation, and W1 and W2 represent the first two weights of the learnable inter-view adapter.
[0035] Through this inter-view aggregation, features from multiple angles are fused into a single representation, obtaining global features of the point cloud from multiple views.
[0036] Step 4-3: Based on compact global features, generate adaptive features f for each view image using a learnable inter-view adapter. i a And added to the initial encoded feature f via residual connections. v The final multi-view encoding feature f is obtained from this. e ;
[0037] After obtaining the compact global features of the point cloud in the previous step, adaptive features for each point cloud are generated using the global features.
[0038]
[0039] Among them, W 3i Let be the i-th view in the third layer of the learnable view adapter, and let W3 be the weight of the third layer of the learnable view adapter. This represents the weights of the 1st, 2nd, and Mth views, and incorporates the globally guided adaptive features into f. i In the middle, f i This indicates that the ResNet network is used to extract features from multi-view images.
[0040] Here, the features obtained through the inter-view adapter are combined with the original features to gain a comprehensive understanding of the point cloud, thereby achieving better view prediction.
[0041] The learnable view adapter is a three-layer multilayer perceptron attached to PointCLIP.
[0042] A learnable inter-view adapter is introduced to obtain more CLIP-adaptive multi-view features, enabling better feature extraction from multiple views in few-sample settings.
[0043] The pre-trained CLIP obtains text features of point clouds with known categories and multi-view features of point clouds to be classified. The text features and multi-view features are compared for similarity, and the category corresponding to the text feature with the highest similarity is used as the text category of the multi-view image features to predict the point cloud category.
[0044] The pre-trained CLIP obtains text features from point clouds of known categories. These text features are then compared with the multi-view features of the point cloud to be classified. The category corresponding to the text feature with the highest similarity is used as the text category of the multi-view image features. This involves placing K point cloud text category names in the class label position of a predefined template: "point cloud depthmapofa[CLASS]". After generating text prompts, the CLIP text encoder extracts features from the text prompts and uses these text features as the CLIP classifier W. t ∈R K×C The input, W t Each K-row vector in the array encodes pre-trained class weights. Simultaneously, CLIP's visual encoder encodes the features of each test image (i.e., the projected view) into f. e ∈R 1 ×K R represents the real number space, K represents the number of text categories, and the classification is calculated as follows:
[0045] logits = f e W t T
[0046] p i =softmax i (logits)
[0047] Where logits represents an intermediate variable, W t T Representing text features, softmax i (·) and p i This represents the softmax function and predicted probability for category i.
[0048] Furthermore, the training of the learnable inter-view adapter freezes the visual and text encoders of CLIP, and the parameters of the learnable inter-view adapter are fine-tuned using cross-entropy loss.
[0049] Furthermore, the smooth loss value calculated by the matching degree between the text prediction category and the true category of the point cloud is used to perform overall iterative update training on the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module and ResNet network.
[0050]
[0051] Where L represents the smooth loss value, exp(y) is the probability that point cloud sample i belongs to the j-th class, and y i y is the true class value of point cloud sample i, y is the predicted class value, and C is the number of point cloud classes.
[0052] Furthermore, the gradient of the overall network parameters is calculated using smooth loss, and the gradient descent method is used to update the overall network parameters.
[0053] Secondly, a classification system employing the aforementioned CLIP-based few-shot classification method for 3D point clouds includes:
[0054] Point cloud angle category tensor acquisition module: The PointNet point cloud classification network is used to extract features from point cloud data, and the point cloud angle category tensor is obtained based on the extracted features.
[0055] Rotation matrix acquisition module: Converts the point cloud angle category tensor obtained by the point cloud angle category tensor acquisition module into Euler angles, and then converts the Euler angles into rotation matrices;
[0056] Multi-view image feature extraction module: The obtained rotation matrix is used to rotate the corresponding point cloud, and the rotated point cloud is subjected to perspective projection using the perspective projection module to obtain multi-view images. The ResNet network is used to extract the features of the multi-view images.
[0057] Point cloud category prediction module: Based on the known category point cloud, a text preset template is set to generate a known point cloud text prompt; the known category point cloud text prompt and the multi-view image features obtained in step 3 are input into the pre-trained CLIP, and the predicted category of the point cloud is output;
[0058] Before point cloud category prediction, the training module uses the point cloud data sample set and the corresponding point cloud category labels to sequentially call the point cloud angle category tensor acquisition module, rotation matrix acquisition module, multi-view image feature extraction module, and point cloud category prediction module for processing. The parameters of the sequentially connected point cloud classification network PointNet, perspective projection module, and ResNet network are trained iteratively as a whole.
[0059] Thirdly, a computer-readable storage medium storing a computer program that is invoked by a processor for execution:
[0060] The steps of the CLIP-based 3D point cloud few-shot classification method described above.
[0061] Fourthly, an electronic terminal includes at least:
[0062] One or more processors;
[0063] A memory that stores one or more computer programs;
[0064] The processor invokes the computer program to execute:
[0065] The steps of the CLIP-based 3D point cloud few-shot classification method described above.
[0066] Beneficial effects
[0067] This invention provides a CLIP-based method and system for few-shot classification of 3D point clouds. The method uses point cloud data and text descriptions as raw inputs, projects the point cloud from multiple angles to obtain multi-view images, and uses these multi-view images and text descriptions as inputs to a pre-trained CLIP model to obtain the corresponding text categories for each multi-view image, thereby obtaining the category of the original point cloud corresponding to the multi-view image. First, a learnable projection module is used to obtain several different optimal projection angles for the point cloud, and then the rotation matrix of the point cloud is obtained using these projection angles. Next, perspective projection is used to obtain the multi-view images after projection. Figure 2The method uses a ResNet network to extract multi-view image features from 3D point cloud images. Finally, it encodes these features using the CLIP visual encoder, extracting text features based on preset categories and templates. This allows for the calculation of scores for features from each viewpoint, followed by weighted summation to obtain the final classification score. This point cloud few-shot classification learning method of the present invention has significant advantages such as simple training and strong versatility. Specifically, the method combines multi-view learning, text description, pre-trained models, few-shot learning, and end-to-end training techniques to reduce reliance on large amounts of 3D labeled data. It transfers the knowledge of the pre-trained CLIP model to new classification tasks and fully utilizes the powerful capabilities of the CLIP model in vision-language joint learning, simplifying the training process. Furthermore, this method can adapt to various different 3D point cloud classification tasks, thus exhibiting strong versatility.
[0068] Compared with existing methods, the advantages of this invention are reflected in:
[0069] 1) A learnable projection angle module is proposed. By transforming the angle regression task into an angle classification task, different optimal projection angles for different objects can be obtained, which solves the problems of "fixed projection angle, optimal projection angles for different objects, and repetitive projection views".
[0070] 2) This invention proposes a method and system for few-shot and zero-shot classification of 3D point clouds based on the large-scale image-text pair pre-trained model CLIP. It utilizes a 2D network to achieve few-shot and zero-shot classification of point clouds and realizes cross-modal learning. Attached Figure Description
[0071] Figure 1 This is a schematic diagram of the execution flow of the method described in the technical solution of the present invention;
[0072] Figure 2 This is a general block diagram of the method described in the technical solution of the present invention. Detailed Implementation
[0073] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments:
[0074] Example:
[0075] This example provides a CLIP-based method for few-shot classification of 3D point clouds. The overall flowchart is as follows: Figure 1 As shown:
[0076] Step 1: Use PointNet, a point cloud classification network, to extract features from the point cloud data and obtain the point cloud angle category tensor based on the extracted features.
[0077] Step 2 converts the point cloud angle category tensor obtained in Step 1 into Euler angles, and then converts the Euler angles into rotation matrices;
[0078] Step 3: Rotate the corresponding point cloud using the obtained rotation matrix, perform perspective projection on the rotated point cloud using the perspective projection module to obtain a multi-view image, and extract the features of the multi-view image using the ResNet network.
[0079] Step 4 sets a text preset template based on the known category point cloud and generates a known point cloud text prompt; input the known category point cloud text prompt and the multi-view image features obtained in Step 3 into the pre-trained CLIP and output the predicted category of the point cloud;
[0080] Before classification, the point cloud data sample set and the corresponding point cloud category labels are processed according to steps 1 to 4, and the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module and ResNet network are trained iteratively as a whole.
[0081] The above steps of the present invention will be described in detail below through an application example.
[0082] The point cloud dataset used in this embodiment is the officially released ModelNet40 dataset. The ModelNet40 dataset consists of point cloud models hand-drawn in CAD and is a large-scale dataset widely used for 3D shape recognition and understanding. The models in the dataset cover common object categories, such as tables, chairs, beds, sofas, bookshelves, monitors, etc., and contain a total of 40 different categories. After the network is trained, it is tested, and the experimental results are given.
[0083] Step 1 uses the PointNet point cloud classification network to extract features from the point cloud data, and obtains the point cloud angle category tensor based on the extracted features, including the following:
[0084] First, download the ModelNet dataset and input it into the PointNet point cloud classification network for feature extraction. Set the output class shape of the classification network to 6×36. It should be noted that the number of views can be changed according to the corresponding task. In this example, 6 views are used to describe a point cloud target. 36 represents the angle class tensor, which corresponds to the angle class tensor in step 2.
[0085] Step 2: Convert the point cloud angle category tensor obtained in Step 1 into Euler angles, and then convert the Euler angles into a rotation matrix. This includes the following sub-steps:
[0086] Step 2-1 converts the point cloud angle category tensor to Euler angles. The specific conversion process is as follows:
[0087] Divide 360° into 36 angle intervals, which are 36 category tensors. The value of the category tensor ranges from 0 to 35, corresponding to the output obtained in step 1. The center value of the angle interval in which the point cloud angle category tensor is located is taken as the center value of the interval in which the Euler angle value of the point cloud angle category tensor is located, and the corresponding Euler angle is taken as the calculation formula as follows:
[0088]
[0089] in, It is the Euler angle of the point cloud around the XYZ directions, N x N y N z This is to obtain the point cloud angle category tensor in the x, y, z directions using the features extracted from the point cloud by the PointNet point cloud classification network;
[0090] For example, if the class tensor is 0, the corresponding Euler angle is 5°; if the class tensor is 1, the corresponding Euler angle is 15°, and so on.
[0091] Step 2-2: Construct the point cloud rotation matrix based on the Euler angles of the point cloud. The construction process is as follows:
[0092]
[0093]
[0094]
[0095]
[0096] Where Rotmat is the rotation matrix of the constructed point cloud. R y (β), R z (γ) is the rotation matrix of the point cloud around the X, Y, and Z directions.
[0097] Step 3: Rotate the corresponding point cloud using the obtained rotation matrix, and then perform perspective projection on the rotated point cloud using the perspective projection module to obtain multi-view images. In this example, six images are used to describe one point cloud object. The ResNet network is used to extract features from the multi-view images, including the following sub-steps:
[0098] Step 3-1: Multiply the input point cloud data with the rotation matrix to rotate the point cloud.
[0099] Objects present different characteristics when viewed from different angles. By performing perspective projection, we can obtain an angular view that best expresses the characteristics of the object, thereby obtaining the view image features that best reflect the characteristics of the object and achieving the most accurate classification.
[0100] Step 3-2: Perform perspective projection on the rotated point cloud.
[0101] Each point cloud is projected onto a series of predefined image planes to generate a scattering depth map, preserving the original attributes of the point cloud from multiple views. The projection process is as follows: Let (x, y, z) be the coordinates of a point p in the point cloud relative to the camera. Perspective projection is applied to obtain the two-dimensional coordinates of p at depth z (x1 = x / z, y1 = y / z). Multiple points can be projected to the same discrete positions on the image planes. To generate depth values at each image position, a weighted average depth is calculated, with greater weight given to points closer to the view. This results in a multi-view image after perspective projection.
[0102] Step 3-3: Use the ResNet network to extract features from the projected multi-view images.
[0103] The acquired multi-view images are input into the ResNet network to obtain the final features.
[0104] Step 4: Based on the known category point cloud of the ModelNet dataset, set the text preset template to generate known point cloud text prompts; input the known category point cloud text prompts and the multi-view image features obtained in Step 3 into the pre-trained CLIP, and output the predicted category of the point cloud;
[0105] The pre-trained CLIP obtains text features of point clouds with known categories and multi-view features of point clouds to be classified. The text features and multi-view features are compared for similarity, and the category corresponding to the text feature with the highest similarity is used as the text category of the multi-view image features to predict the point cloud category.
[0106] Before classification, the point cloud data sample set and corresponding point cloud category labels are processed according to steps 1 to 4, and the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module, and ResNet network are trained iteratively. This includes the following sub-steps:
[0107] Step 4-1: Encode the multi-view features of the point cloud to be classified using the visual encoder in the pre-trained CLIP to obtain the initial encoded features f. v f v ∈R 1×C ;
[0108] C represents a feature dimension of 128, and R represents the real number space;
[0109] Step 4-2: Using a learnable inter-view adapter, the features encoded in Step 4-1 are concatenated to obtain compact global features of the point cloud;
[0110] Specifically, given six CLIP-encoded multi-view features of a point cloud, they are concatenated into a feature called Concate(f). 1-6 ), Concate(f 1-6 )∈R 1×6C And obtain compact global features of the point cloud through the first two layers of the learnable inter-view adapter;
[0111]
[0112] Among them, f global f represents the compact global features of a point cloud. global ∈R 1×C Concate(f) 1-6 ) represents the features after connecting the features from view 1 to 6, Concate represents the feature connection operation, and W1 and W2 represent the first two layers of weights of the learnable inter-view adapter.
[0113] Through this inter-view aggregation, features from multiple angles are fused into a single representation, obtaining global features of the point cloud from multiple views.
[0114] Step 4-3: Based on compact global features, generate adaptive features f for each view image using a learnable inter-view adapter. i a And added to the initial encoded feature f via residual connections. v The final multi-view encoding feature f is obtained from this. e ;
[0115] After obtaining the compact global features of the point cloud in the previous step, adaptive features for each point cloud are generated using the global features.
[0116]
[0117] Among them, W 3i Let be the i-th view in the third layer of the learnable view adapter, and let W3 be the weight of the third layer of the learnable view adapter. This represents the weights of the 1st, 2nd, and 6th views, and incorporates the globally guided adaptive features into f. i In the middle, f i This indicates that the ResNet network is used to extract features from multi-view images.
[0118] Here, the features obtained through the inter-view adapter are combined with the original features to gain a comprehensive understanding of the point cloud, thereby achieving better view prediction.
[0119] The learnable view adapter is a three-layer multilayer perceptron attached to PointCLIP.
[0120] A learnable inter-view adapter is introduced to obtain more CLIP-adaptive multi-view features, enabling better feature extraction from multiple views in few-sample settings.
[0121] The pre-trained CLIP obtains text features of point clouds with known categories and multi-view features of point clouds to be classified. The text features and multi-view features are compared for similarity, and the category corresponding to the text feature with the highest similarity is used as the text category of the multi-view image features to predict the point cloud category.
[0122] Step 4-4: Obtain text features for point cloud classification
[0123] The pre-trained CLIP obtains text features from point clouds of known categories. These text features are then compared with the multi-view features of the point cloud to be classified. The category corresponding to the text feature with the highest similarity is used as the text category of the multi-view image features. This means placing the K point cloud text category names in the class label position of a predefined template: "point cloud depthmapofa[CLASS]". After generating text prompts, the CLIP text encoder extracts features from the text prompts, and these text features are used as the CLIP classifier W. t ∈R K×C The input, W t Each of the K row vectors in the array encodes pre-trained class weights.
[0124] Steps 4-5: Build a classifier to obtain text matching predictions
[0125] The features of each test image, i.e., the projected view, are encoded into f using the CLIP visual encoder. e ∈R 1×K R represents the real number space, and K represents the number of text categories. In this example, K is 40, and the category calculation is as follows:
[0126] logits = f e W t T
[0127] p i =softmax i (logits)
[0128] Where logits represents an intermediate variable, W t T Representing text features, softmax i (·) and p i This represents the softmax function and predicted probability for category i.
[0129] The specific process of using CLIP to classify point cloud multi-views in this embodiment is as follows:
[0130] (1) Training the learnable inter-view adapter, freezing CLIP’s visual and text encoders, and fine-tuning the parameters of the learnable inter-view adapter by cross-entropy loss.
[0131] (2) The smooth loss value calculated by the matching degree between the text prediction category and the real category of the point cloud is used to perform overall iterative update training on the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module and ResNet network.
[0132]
[0133] Where L represents the smooth loss value, exp(y) is the probability that point cloud sample i belongs to the j-th class, and y i y is the true class value of point cloud sample i, y is the predicted class value, and C is the number of point cloud classes.
[0134] (3) Calculate the parameter gradient of the entire network using smooth loss, and update the parameters of the entire network using gradient descent.
[0135] (4) Set the training epoch of the whole network to 250 times, and save the weight with the best classification accuracy. In this training example, the weight with the best classification accuracy is when epoch=245, and the average classification accuracy of all categories is 89.3%. After training, test the whole network.
[0136] (5) The entire network was tested based on the weights of the saved epoch=245. The number of test samples was 2468, of which 2179 were correctly classified and the average classification accuracy was 88.31%.
[0137] This invention also provides a classification system employing the above-described CLIP-based few-shot classification method for 3D point clouds, comprising:
[0138] Point cloud angle category tensor acquisition module: The PointNet point cloud classification network is used to extract features from point cloud data, and the point cloud angle category tensor is obtained based on the extracted features.
[0139] Rotation matrix acquisition module: Converts the point cloud angle category tensor obtained by the point cloud angle category tensor acquisition module into Euler angles, and then converts the Euler angles into rotation matrices;
[0140] Multi-view image feature extraction module: The obtained rotation matrix is used to rotate the corresponding point cloud, and the rotated point cloud is subjected to perspective projection using the perspective projection module to obtain multi-view images. The ResNet network is used to extract the features of the multi-view images.
[0141] Point cloud category prediction module: Based on the known category point cloud, a text preset template is set to generate a known point cloud text prompt; the known category point cloud text prompt and the multi-view image features obtained in step 3 are input into the pre-trained CLIP, and the predicted category of the point cloud is output;
[0142] Before point cloud category prediction, the training module uses the point cloud data sample set and the corresponding point cloud category labels to sequentially call the point cloud angle category tensor acquisition module, rotation matrix acquisition module, multi-view image feature extraction module, and point cloud category prediction module for processing. The parameters of the sequentially connected point cloud classification network PointNet, perspective projection module, and ResNet network are trained iteratively as a whole.
[0143] For details on the implementation process of each module, please refer to the methods described above; they will not be repeated here. It should be understood that the above division of functional modules is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the integrated units described above can be implemented in hardware or as software functional units.
[0144] This invention also provides a computer-readable storage medium storing a computer program that is invoked by a processor for execution:
[0145] The steps of the CLIP-based 3D point cloud few-shot classification method described above.
[0146] Please refer to the explanation of the method above for the specific implementation process of each step.
[0147] This invention also provides an electronic terminal, comprising at least:
[0148] One or more processors;
[0149] A memory that stores one or more computer programs;
[0150] The processor invokes the computer program to execute:
[0151] The steps of the CLIP-based 3D point cloud few-shot classification method described above.
[0152] Please refer to the explanation of the method above for the specific implementation process of each step.
[0153] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the controller described in any of the foregoing embodiments, such as the controller's hard drive or memory. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the controller. Further, the readable storage medium can include both the controller's internal storage unit and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.
[0154] Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0155] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.
[0156] It should be emphasized that the examples described in this invention are illustrative rather than limiting. Therefore, this invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of this invention, without departing from the spirit and scope of this invention, whether modifications or substitutions, are also within the protection scope of this invention.
Claims
1. A few-shot classification method for 3D point clouds based on CLIP, characterized in that, include: Step 1: Use the PointNet point cloud classification network to extract features from the point cloud data, and obtain the point cloud angle category tensor based on the extracted features; Step 2: Convert the point cloud angle category tensor obtained in Step 1 into Euler angles, and then convert the Euler angles into rotation matrices; Step 3: Rotate the corresponding point cloud using the obtained rotation matrix, perform perspective projection on the rotated point cloud using the perspective projection module to obtain multi-view images, and extract multi-view image features using the ResNet network. Step 4: Set a text preset template based on the known category point cloud to generate a known point cloud text prompt; input the known category point cloud text prompt and the multi-view image features obtained in Step 3 into the pre-trained CLIP, and output the predicted category of the point cloud; Before classification, the point cloud data sample set and the corresponding point cloud category labels are processed according to steps 1 to 4, and the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module and ResNet network are trained iteratively as a whole. The process of inputting the known category point cloud text prompts and the multi-view image features obtained in step 3 into the pre-trained CLIP to obtain the final multi-view encoded features of the point cloud to be classified is as follows: Step 4-1: Encode the multi-view features of the point cloud to be classified using the visual encoder in the pre-trained CLIP to obtain the initial encoded features. , ; C represents the feature dimension, and R represents the real number space; Step 4-2: Using a learnable inter-view adapter, the features encoded in Step 4-1 are concatenated to obtain compact global features of the point cloud; Specifically, given M CLIP-encoded multi-view features of a point cloud, these features are concatenated into a single feature set. , And obtain compact global features of the point cloud through the first two layers of the learnable inter-view adapter; ； in, Represents the compact global features of a point cloud. , This represents the features obtained by connecting the features of views 1 to M, where M is the number of views. Indicates feature connection operation, , This represents the weights of the first two layers of the learnable inter-view adapter; Step 4-3: Based on compact global features, generate adaptive features for each view image using a learnable inter-view adapter. And added to the initial encoded features via residual connections. The final multi-view encoding features are obtained from this process. ; ； in, Let i be the i-th view in the third layer of the learnable inter-view adapter, and let the weights of the third layer of the learnable inter-view adapter be... , =[ ; ; ] , , , This represents the weights of the 1st, 2nd, and Mth views, and incorporates the globally guided adaptive features. middle, This indicates that the ResNet network is used to extract features from multi-view images; The learnable inter-view adapter is a three-layer multilayer perceptron attached to the PointCLIP; The learnable inter-view adapter is trained by freezing CLIP's visual and text encoders and fine-tuning the parameters of the learnable inter-view adapter using cross-entropy loss.
2. The method according to claim 1, characterized in that, The process of obtaining the rotation matrix is as follows: Step 2-1: Convert the point cloud angle category tensor to Euler angles; Divide 360° into 36 angle intervals, which are 36 category tensors. The value of the category tensor ranges from 0 to 35. The center value of the angle interval in which the point cloud angle category tensor is located is taken as the center value of the interval in which the corresponding Euler angle value of the point cloud angle category tensor is located. The calculation formula is as follows: ， ， ； in, It is the Euler angle of the point cloud around the XYZ directions. , , This is to obtain the point cloud angle category tensor in the x, y, z directions using the features extracted from the point cloud by the PointNet point cloud classification network; Step 2-2: Construct the point cloud rotation matrix based on the Euler angles of the point cloud; ； ； ； ； in, For the constructed point cloud rotation matrix, , , Let be the rotation matrix of the point cloud around the X, Y, and Z directions.
3. The method according to any one of claims 1-2, characterized in that, The smooth loss value, calculated by matching the text prediction category and the true category of the point cloud, is used to iteratively update the parameters of the sequentially connected point cloud classification network PointNet, perspective projection module and ResNet network. ； in, Let represent the smooth loss value, and exp(y) be the probability that point cloud sample i belongs to the j-th class. It is the true class value of point cloud sample i. It is the predicted category value. It represents the number of point cloud categories.
4. The method according to claim 3, characterized in that, The gradient of the overall network parameters is calculated using smooth loss, and the gradient descent method is used to update the overall network parameters.
5. A classification system employing the CLIP-based few-shot classification method for 3D point clouds as described in any one of claims 1-4, characterized in that, include: Point cloud angle category tensor acquisition module: The PointNet point cloud classification network is used to extract features from the point cloud data, and the point cloud angle category tensor is obtained based on the extracted features. Rotation matrix acquisition module: Converts the point cloud angle category tensor obtained by the point cloud angle category tensor acquisition module into Euler angles, and then converts the Euler angles into rotation matrices; Multi-view image feature extraction module: The obtained rotation matrix is used to rotate the corresponding point cloud, and the rotated point cloud is subjected to perspective projection using the perspective projection module to obtain multi-view images. The ResNet network is used to extract the features of the multi-view images. Point cloud category prediction module: Based on known category point clouds, a text preset template is set to generate known point cloud text prompts; the known category point cloud text prompts and the obtained multi-view image features are input into the pre-trained CLIP, and the predicted category of the point cloud is output; Before point cloud category prediction, the training module uses the point cloud data sample set and the corresponding point cloud category labels to sequentially call the point cloud angle category tensor acquisition module, rotation matrix acquisition module, multi-view image feature extraction module, and point cloud category prediction module for processing. The parameters of the sequentially connected point cloud classification network PointNet, perspective projection module, and ResNet network are trained iteratively as a whole.
6. A computer-readable storage medium, characterized in that: The computer program is stored and is invoked by the processor for execution. The steps of the CLIP-based three-dimensional point cloud few-shot classification method according to any one of claims 1-4.
7. An electronic terminal, characterized in that: At least includes: One or more processors; A memory that stores one or more computer programs; The processor invokes the computer program to execute: The steps of the CLIP-based three-dimensional point cloud few-shot classification method according to any one of claims 1-4.