Point cloud quantification method and system based on multi-modal prompt, and medium
By employing multimodal cues and the Gumbel-Softmax quantization strategy, the problem of aligning point cloud features with text prototypes was solved, enabling efficient quantization of point cloud features and cross-modal feature fusion, thereby improving the accuracy and robustness of point cloud analysis.
Patent Information
- Application Number
- CN202511330820.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-17
- Publication Date
- 2025-12-19
AI Technical Summary
In point cloud analysis tasks, visual encoders mainly focus on the surface geometry of objects, while text encoders are more inclined to abstract semantic understanding. This makes it difficult for point cloud features to establish an accurate correspondence with the text prototype, affecting the generalization ability and quantization effect of point cloud features.
A multimodal prompting approach is adopted, which uses a pre-trained ULIP model to extract features from text and point cloud inputs respectively. The point cloud features are quantized by the Gumbel-Softmax reparameterization method, and cross-attention mechanism and loss function optimization are introduced to achieve accurate alignment and fusion of point cloud features and text prototypes.
It improves the discriminative and generalization capabilities of point cloud features, alleviates the multimodal semantic gap problem, and enhances the robustness and accuracy of point cloud representation, especially under conditions of small sample size and cross-dataset performance.
Smart Images
Figure CN121168465A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of computer vision and point cloud recognition, and particularly relates to a point cloud quantization method and system based on multi-modal prompts and a medium. BACKGROUND
[0002] In urban air traffic, low-altitude unmanned aerial vehicle inspection and autonomous flight applications, three-dimensional point cloud data as an important means of perceiving environmental structure is crucial for achieving high-precision spatial understanding and navigation control. Efficient representation and understanding of point cloud data have core value for tasks such as three-dimensional environment mapping, obstacle recognition and path planning. Traditional point cloud quantization methods mainly rely on vector quantization, including prototype learning based on clustering centers and codebook training. However, these methods usually have great limitations, for example, the prototype method based on clustering relies heavily on known data, resulting in weak generalization ability when processing new data, while the codebook-based method can generate trainable prototypes, but its network training is complex and prone to converge to suboptimal solutions, thereby affecting the quantization effect and feature expression ability.
[0003] In recent years, large multi-modal models have made significant progress in the field of computer vision and natural language processing, especially models such as CLIP and ULIP successfully align visual and language features through contrastive learning, thereby improving zero-shot learning ability. However, in the point cloud analysis task, since the visual encoder mainly focuses on the surface geometric information of objects, while the text encoder is more inclined to abstract semantic understanding, there is still a large semantic gap between the two, making it difficult for point cloud features to directly establish an accurate correspondence with text prototypes. SUMMARY
[0004] The application provides a point cloud quantization method and system based on multi-modal prompts and a medium, aiming to solve the problem that in the point cloud analysis task, since the visual encoder mainly focuses on the surface geometric information of objects, while the text encoder is more inclined to abstract semantic understanding, there is still a large semantic gap between the two, making it difficult for point cloud features to directly establish an accurate correspondence with text prototypes.
[0005] The application is implemented as follows: a point cloud quantization method based on multi-modal prompts, comprising the following steps:
[0006] S1, feature extraction: using a pre-trained multi-modal large model to extract features from text input and point cloud input respectively, wherein the text input generates semantic features through a ULIP text encoder, and the point cloud input extracts point cloud features through a ULIP three-dimensional point cloud encoder;
[0007] S2, point cloud quantization: text features are used to construct class prototypes, which are used for quantization of point cloud features. The point cloud feature quantization adopts the Gumbel-Softmax reparameterization method, and its quantization process can be represented as:
[0008]
[0009] S3, feature fusion: the quantized point cloud features are fused with the original point cloud features. The fusion process is realized through cross-attention mechanism and feedforward neural network. The specific calculation formula is as follows:
[0010]
[0011] Preferably, in S2, y ik is the discrete probability distribution calculated based on Gumbel-Softmax:
[0012]
[0013] Where, ∈ k Gumbel(0, 1) is Gumbel noise, and τ is a temperature parameter used to control the smoothness of sampling.
[0014] Preferably, in S3, is the point cloud feature, q i is the quantized feature, and the final fused feature f i is used as the input of the classification task.
[0015] Preferably, in S2, the intra-class loss and inter-class loss are introduced in the point cloud feature quantization process, wherein the intra-class loss is used to reduce the distance between the point cloud features of the same class and the prototype, and the calculation formula is as follows:
[0016]
[0017] Where, N is the number of point cloud samples in the batch, represents the feature of the i-th point cloud, q i is the quantized point cloud feature, and the inter-class loss is used to maximize the distance between the prototypes of different classes:
[0018]
[0019] Where, and are text prototypes of different classes, and τ is a hyperparameter;
[0020] The final loss function is composed of contrastive loss, intra-class loss and inter-class loss, and the calculation formula is as follows:
[0021] L total = Lalign +λ1L intra +λ2L inter
[0022] where λ1 and λ2 are hyperparameters.
[0023] Preferably, in S1, the text prototype is extracted by the ULIP text encoder and used to construct the category prototype, the text input is enhanced by the learnable prompt, and the generated text feature is defined as:
[0024]
[0025] where F T (·) represents the text encoder, u1, u2,..., u m is the learnable prompt vector, c k is the category text embedding.
[0026] Preferably, in S2, the point cloud feature is extracted by the ULIP three-dimensional point cloud encoder, and the extraction of the point cloud feature is defined as:
[0027] where F′ P is the fine-tuned ULIP three-dimensional point cloud encoder, P i is the input point cloud, and N is the total number of point cloud samples.
[0028] Preferably, in S2, the probability calculation method of the quantization process is:
[0029]
[0030] where s ik is the cosine similarity between the point cloud feature and the prototype feature:
[0031]
[0032] Preferably, in S3, the fused final feature is used for classification tasks and optimized by contrastive loss to further align the point cloud feature and the text prototype, and the calculation formula is as follows:
[0033]
[0034] where f i represents the fused point cloud feature, is the category prototype corresponding to the fused feature.
[0035] A point cloud quantization system based on multi-modal prompts, comprising a text input module, a point cloud input module, a prototype generation module, a point cloud quantization module, a feature fusion module, and a classification recognition module.
[0036] The text input module is used for receiving text input and extracting text features through a text encoder; the point cloud input module is used for receiving point cloud data and extracting point cloud features through a point cloud encoder; the prototype generation module is used for constructing a category prototype based on the text features, and optimizing the prototype by using a learnable prompt; the point cloud quantization module is used for quantizing the point cloud features to the prototype feature space by using the Gumbel-Softmax technology; the feature fusion module is used for fusing the quantized point cloud features and the original point cloud features; and the classification recognition module is used for classifying the point cloud based on the fused features, so as to improve the accuracy and generalization ability of point cloud recognition.
[0037] A computer-readable storage medium stores a computer program, the computer program comprising program instructions, the program instructions being executed by a processor to cause the processor to perform the method of any one of the above
[0038] Advantages
[0039] Compared with the prior art, the advantages of the present application are:
[0040] (1) The present application proposes a point cloud quantization method based on multi-modal prompt driving, which uses the text features in the pre-trained language-visual model as semantic prototypes to guide the quantization process of point cloud features, realizes the mapping of point cloud features to an interpretable semantic space, and thus improves the discriminability and generalization ability of point cloud representation.
[0041] (2) The present application designs a learnable prompt mechanism, which performs adaptive optimization on the prototype representation, introduces compactness and separability constraints, and can adaptively adjust the semantic alignment degree between visual and language modalities, significantly alleviates the multi-modal semantic gap problem, and avoids the instability problem of prototype training in traditional clustering or codebook methods.
[0042] (3) The present application proposes a differentiable discrete quantization strategy based on Gumbel-Softmax reparameterization, which preserves the discrete semantic structure while maintaining the differentiability of the quantization process, effectively fuses the geometric details of the point cloud and the high-order semantic information in the language prototype, and further improves the robustness and accuracy under the conditions of small sample and cross-dataset. BRIEF DESCRIPTION OF DRAWINGS
[0043] Figure 1 The flowchart of the point cloud quantization method based on multi-modal prompt driving of the present application;
[0044] Figure 2 The schematic diagram of the overall network structure of the present application;
[0045] Figure 3 The full-supervised experimental result graph on the ModelNet40 and ScanObjectNN data sets;
[0046] Figure 4 Figures for few-shot learning experiment results on ScanObjectNN-PB dataset;
[0047] Figure 5 Figures for 4-shot few-shot experiment results on ScanObjectNN dataset. DETAILED DESCRIPTION
[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.
[0049] Please refer to Figures 1-2 The present application provides a technical solution: a point cloud quantization method based on multi-modal prompt, comprising the following steps:
[0050] S1, feature extraction: using a pre-trained multi-modal large model to extract features from text input and point cloud input respectively, wherein the text input generates semantic features through a ULIP text encoder, and the point cloud input extracts point cloud features through a ULIP three-dimensional point cloud encoder;
[0051] S2, point cloud quantization: the text features are used to construct a category prototype, and the prototype is used for quantization of the point cloud features, and the point cloud feature quantization adopts a Gumbel-Softmax reparameterization method, and the quantization process can be represented as:
[0052]
[0053] S3, feature fusion: the quantized point cloud features and the original point cloud features are fused, and the fusion process is realized through a cross-attention mechanism and a feedforward neural network, and the specific calculation formula is as follows:
[0054]
[0055] In the S2, y ik is a discrete probability distribution calculated based on Gumbel-Softmax:
[0056]
[0057] wherein, ∈ k Gumbel(0, 1) is Gumbel noise, and τ is a temperature parameter for controlling the smoothness of sampling.
[0058] In the S3, is the point cloud feature, q i is the quantized feature, and the final fused feature fi as the input of the classification task.
[0059] Further, in the S2, an intra-class loss and an inter-class loss are introduced in the point cloud feature quantization process, wherein the intra-class loss is used to reduce the distance between the point cloud features of the same class and the prototype, and the calculation formula is as follows:
[0060]
[0061] wherein N is the number of point cloud samples in a batch, denotes the feature of the i-th point cloud, q i is the quantized point cloud feature, and the inter-class loss is used to maximize the distance between the prototypes of different classes:
[0062]
[0063] wherein, and are text prototypes of different classes, and τ is a hyperparameter;
[0064] The final loss function is composed of the contrastive loss, the intra-class loss and the inter-class loss, and the calculation formula is as follows:
[0065] L total = L align + λ1L intra + λ2L inter
[0066] wherein λ1 and λ2 are hyperparameters.
[0067] Further, in the S1, the text prototype is extracted by the ULIP text encoder and is used to construct the class prototype, and the text input is enhanced by the learnable prompt, and the generated text feature is defined as:
[0068]
[0069] wherein F T (·) denotes the text encoder, u1, u2,..., u m are learnable prompt vectors, and c k is the class text embedding.
[0070] Further, in the S2, the point cloud feature is extracted by the ULIP three-dimensional point cloud encoder, and the extraction of the point cloud feature is defined as:
[0071] wherein F′ P is the fine-tuned ULIP three-dimensional point cloud encoder, P i is the input point cloud, and N is the total number of point cloud samples.
[0072] Further, in the S2, the probability calculation method of the quantization process is:
[0073]
[0074] where s ik is the cosine similarity between the point cloud feature and the prototype feature:
[0075]
[0076] Further, in the S3, the final feature after fusion is used for classification task, and contrastive loss is used for optimization to further align the point cloud feature and the text prototype, and the calculation formula is as follows:
[0077]
[0078] where f i represents the fused point cloud feature, is the class prototype corresponding to the fused feature.
[0079] A point cloud quantization system based on multi-modal prompt, comprising a text input module, a point cloud input module, a prototype generation module, a point cloud quantization module, a feature fusion module and a classification recognition module;
[0080] The text input module is used for receiving text input and extracting text features through a text encoder; the point cloud input module is used for receiving point cloud data and extracting point cloud features through a point cloud encoder; the prototype generation module is used for constructing class prototypes based on text features and optimizing the prototypes using learnable prompts; the point cloud quantization module is used for quantizing point cloud features to prototype feature space using Gumbel-Softmax technology; the feature fusion module is used for fusing the quantized point cloud features and the original point cloud features; and the classification recognition module is used for classifying point clouds based on the fused features, thereby improving the accuracy and generalization ability of point cloud recognition.
[0081] A computer-readable storage medium stores a computer program, the computer program comprising program instructions, the program instructions being executed by a processor to cause the processor to execute the method of any one of the above.
[0082] Embodiment 1
[0083] This invention employs the ULIP multimodal pre-trained model to extract features from both text input and point cloud input. The text input generates category prototypes through the ULIP text encoder, and optimizes the category representation through learnable hints, enabling the category prototypes to more accurately represent point cloud category information and narrow the semantic gap between visual and text features. The point cloud input extracts point cloud features through the ULIP point cloud encoder, and these features are used for subsequent quantization processing to ensure that the point cloud can align with the category prototypes and be effectively classified.
[0084] This invention employs the Gumbel-Softmax reparameterization method to perform differentiable quantization of point cloud features, enabling the point cloud features to be effectively mapped to the category prototype space. First, the similarity between the point cloud features and the category prototypes is calculated to generate a category probability distribution. Then, the Gumbel-Softmax method is used to reparameterize the category probabilities, allowing the quantization process to effectively discretize the point cloud features while maintaining differentiability. Unlike traditional hard-coded quantization, this invention's method models differentiable category probability distributions, enabling the quantized features to adapt to different categories of point cloud data distributions, while ensuring that the point cloud features are reasonably assigned to the corresponding category prototypes. Finally, the final quantized point cloud features are obtained through a weighted summation of category prototypes. These features are expressed in the category prototype space for subsequent feature fusion and optimization.
[0085] After quantization, this invention employs a cross-modal attention mechanism to fuse point cloud features with category prototypes, enabling the quantized point cloud features to fully absorb category information and improve the semantic consistency of the point cloud features. First, the correlation between point cloud features and category prototypes is calculated through the attention mechanism, and fusion weights are assigned according to the importance of different category prototypes. Then, the fused features are adjusted through a feedforward neural network to maintain consistency with the feature format of the original output of the point cloud encoder, and to ensure that the fused features have better separability and representational ability in classification tasks.
[0086] To further enhance the classification ability of point cloud features, this invention employs intra-class loss, inter-class loss, and contrastive loss for optimization. Intra-class loss is used to bring point cloud features of the same class as close as possible to their class prototypes, thereby enhancing the compactness of class representation. Inter-class loss is used to maximize the distance between different class prototypes, ensuring good discriminability between classes. Contrastive loss is used to optimize the alignment effect between point cloud features and class prototypes, enabling point cloud features to more effectively match text class representations, thereby improving the model's generalization ability and robustness. The final loss function comprehensively considers the above optimization strategies, further enhancing the expressive power of point cloud features through joint training, and demonstrating superior point cloud recognition performance in experiments.
[0087] In summary, this method constructs a point cloud quantization framework through four modules: text feature extraction, point cloud feature quantization, cross-modal feature fusion, and optimization strategies. While ensuring the differentiability of the quantization process, it improves the expressive power of point cloud features and optimizes the accuracy and robustness of point cloud classification. This method is applicable to various tasks such as autonomous driving, 3D object recognition, and robot perception, and demonstrates superior performance in experiments.
[0088] Example 2
[0089] The scheme in Example 1 will be further described below with specific examples and experimental data:
[0090] I. Data Preparation
[0091] This invention conducts experiments on multiple standard point cloud datasets to verify the effectiveness of the proposed method. The experimental datasets include ModelNet40 and ScanObjectNN. ModelNet40 is currently the most widely used benchmark dataset for point cloud classification, containing synthetic CAD models across 40 classes, totaling 9,843 training samples and 2,468 test samples. ScanObjectNN is a more challenging real-world 3D point cloud classification dataset, containing 2,902 object samples from 15 classes with more complex backgrounds and noise interference. During the experiments, we use three common variants of the ScanObjectNN dataset for testing: OBJ ONLY (objects only), OBJ BG (background included), and PB T50RS (perturbations and random sampling included) to comprehensively evaluate the robustness of the proposed method.
[0092] II. Details on the Use of the Model
[0093] The experiments of this invention were conducted on two NVIDIA A100 Tensor Core GPUs. All models were trained for 250 epochs with a batch size of 30. The optimizer used was AdamW with an initial learning rate of 0.003. A 10-epoch warm-up strategy was employed during training, followed by a cosine annealing learning rate decay strategy to gradually reduce the learning rate, thereby improving training stability and convergence speed. All point cloud data were uniformly sampled to 1024 points to ensure input format consistency. The backbone network of the model used PointBERT as the point cloud encoder, and multimodal pre-trained weights were used with the ULIP framework. During training, all parameters of the visual encoder and text encoder were frozen, and only the text cue vector and the last Transformer block in the point cloud encoder were fine-tuned to reduce computational resource overhead while maintaining the expressive power of cross-modal features. The length of the text cue vector was set to 32, and the hyperparameter during optimization was set to 0.01 to ensure intra-class compactness and inter-class separability, thereby improving the separability and alignment of point cloud features. In addition, to enhance the model's generalization ability, data augmentation strategies such as random rotation, scale variation, and point cloud jitter were incorporated during training to simulate point cloud data deformation in real-world applications and improve the model's robustness to point cloud data noise and deformation.
[0094] III. Evaluation Indicators
[0095] To fairly evaluate the performance of the method of this invention, this experiment uses Top-1 classification accuracy, commonly used in classification tasks, as the main evaluation metric. Simultaneously, in the few-shot experiment, the average accuracy under different sample sizes is used for comparative analysis. Top-1 classification accuracy is defined as follows:
[0096]
[0097] Where N is the total number of test samples, Let y be the predicted category of the i-th sample. i Let be the true class of the sample, and 1(·) be an indicator function that takes the value 1 when the predicted class matches the true class, and 0 otherwise.
[0098] The embodiments of the present invention have the following three key inventive points:
[0099] I. A method for constructing category prototypes based on learnable prompts is proposed.
[0100] Technical effects: This method optimizes the text category prototype by introducing learnable cues, enabling it to more accurately express point cloud category information and narrowing the semantic gap between visual and text features, thereby improving the alignment and classification accuracy of point cloud features.
[0101] II. A point cloud feature quantization strategy based on Gumbel-Softmax is proposed.
[0102] Technical effect: This strategy achieves end-to-end differentiable point cloud discretization while maintaining the expressive power of point cloud features, enabling point cloud features to be adaptively mapped to the most relevant category prototypes, thereby improving the generalization of point cloud classification.
[0103] Third, a cross-modal aligned point cloud feature optimization strategy is proposed.
[0104] Technical effect: By optimizing the class distribution of point cloud features through intra-class loss, inter-class loss and contrast loss, the separability of point cloud features is improved, making them more accurately match text category representations, thereby improving classification performance.
[0105] In summary, this invention provides a point cloud quantization method based on multimodal cues, which combines text category prototype optimization, Gumbel-Softmax quantization, and cross-modal feature fusion to enable point cloud features to be effectively quantized and aligned in cross-modal space, and achieves excellent classification performance and generalization ability on multiple point cloud datasets.
[0106] Example 3
[0107] This invention presents comparative experiments on datasets such as ModelNet40 and ScanObjectNN, comparing it with various point cloud classification methods. The comparison methods include supervised learning-based point cloud classification methods such as PointNet, PointNet++, and DGCNN, as well as pre-trained point cloud classification methods such as PointBERT, MaskPoint, and Point-MAE. Furthermore, this invention is also compared with PPT, a cue-based fine-tuning method based on a multimodal pre-trained large model.
[0108] The fully supervised experimental results on the ModelNet40 and ScanObjectNN datasets are shown in the attached figures in the following description. Figure 3 As shown, the experimental metric is Top-1 classification accuracy. Our method achieved a Top-1 classification accuracy of 90.76%, which is the best among all the compared methods, while maintaining low computational overhead, demonstrating superior performance and computational efficiency.
[0109] The few-shot learning experimental results on the ScanObjectNN-PB dataset are shown in the attached figures in the following specification. Figure 4As shown in the figure, the proposed method achieves Top-1 classification accuracies of 57.1%, 62.3%, 68.5%, and 73.9% for sample sizes K=2, K=4, K=8, and K=16, respectively, all of which are higher than the comparison methods. This result demonstrates that the proposed method maintains strong generalization ability in low-sample learning scenarios, and its recognition accuracy further improves with the increase of training samples, proving the effectiveness of the method in low-resource learning environments.
[0110] Furthermore, embodiments of the present invention compare the performance of different variants of this method on the ScanObjectNN dataset for 4-shot few-shot classification. The experimental results are shown in the accompanying drawings below. Figure 5 As shown, the three tested methods are w / o Point cloud adapter (without fine-tuning the point cloud encoder), w / o Learnable text prompt (using only fixed text prompts instead of learnable text prompts), and w / o Quantization (without feature quantization). The Baseline method does not use text category prototype optimization, the Fixed Prompt method only uses fixed text prompts, such as "a 3D object of a", and the w / o Quantization method does not quantize point cloud features. Under the same training and testing conditions, the Top-1 classification accuracy of the complete method of this invention is higher than all variants, fully verifying the effectiveness of category prototype optimization, quantization strategy, and multimodal feature fusion. This demonstrates that the method of this invention can better utilize cross-modal information, improve the expressive power of point cloud features, and thus achieve better performance in classification tasks.
[0111] This invention addresses point cloud recognition tasks in complex low-altitude environments, and is particularly suitable for environmental modeling needs of low-altitude autonomous vehicles during path planning and obstacle avoidance.
[0112] This invention can be used for high-frequency mapping tasks in low-altitude scenarios, such as intelligent measurement, three-dimensional traffic monitoring, and three-dimensional inspection of substations, to improve the accuracy of point cloud representation and classification stability.
[0113] This invention can be widely applied to the data understanding module of low-altitude work platforms, enhancing their ability to identify irregular three-dimensional structures in situations with few samples, and is suitable for tasks such as bridge inspection and building scanning.
[0114] This invention can be used for three-dimensional target recognition and navigation tasks in urban low-altitude space, such as building facade modeling, aerial obstacle detection and map building.
[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A point cloud quantization method based on multimodal cues, characterized in that: Includes the following steps: S1. Feature Extraction: Features are extracted from text input and point cloud input using a pre-trained multimodal large model. The text input generates semantic features through the ULIP text encoder, and the point cloud input extracts point cloud features through the ULIP 3D point cloud encoder. S2. Point Cloud Quantization: Text features are used to construct category prototypes, which are then used for point cloud feature quantization. The point cloud feature quantization employs the Gumbel-Softmax reparameterization method, and its quantization process can be represented as follows: S3. Feature Fusion: The quantized point cloud features are fused with the original point cloud features. The fusion process is achieved through a cross-attention mechanism and a feedforward neural network. The specific calculation formula is as follows:
2. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In S2, y ik The discrete probability distribution calculated based on Gumbel-Softmax: Where, ∈ k ~Gumbel(0,1) is Gumbel noise, and τ is a temperature parameter used to control the smoothness of the sampling.
3. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In S3, For point cloud features, q i The quantized features are then fused into a final feature f. i As input for a classification task.
4. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In step S2, intra-class loss and inter-class loss are introduced during the point cloud feature quantization process. The intra-class loss is used to reduce the distance between point cloud features of the same class and the prototype. The calculation formula is as follows: Where N is the number of point cloud samples in the batch, Let q represent the feature of the i-th point cloud. i For the quantized point cloud features, the inter-class loss is used to maximize the distance between prototypes of different classes: in, and These represent different categories of text prototypes, and τ is a hyperparameter. The final loss function is a weighted average of the contrastive loss, intra-class loss, and inter-class loss, and is calculated as follows: L total =L align +λ1L intra +λ2L inter Where λ1 and λ2 are hyperparameters.
5. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In step S1, the text prototype is extracted using the ULIP text encoder and used to construct the category prototype. The text input is enhanced with learnable cues, and the generated text features are defined as follows: Among them, F T (·) represents the text encoder, u1, u2, ..., u m For learnable cue vectors, c k Embed the category text.
6. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In step S2, point cloud features are extracted using the ULIP 3D point cloud encoder. The definition of point cloud feature extraction is as follows: Among them, F′ P For fine-tuning the ULIP 3D point cloud encoder, P i The input is a point cloud, and N is the total number of point cloud samples.
7. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In S2, the probability calculation method for the quantization process is as follows: Among them, s ik Cosine similarity between point cloud features and prototype features:
8. The point cloud quantization method based on multimodal cues as described in claim 1, characterized in that: In step S3, the fused final features are used for the classification task and optimized using contrastive loss to further align the point cloud features with the text prototype. The calculation formula is as follows: Among them, f i This represents the features of the fused point cloud. This refers to the category prototype corresponding to the fused features.
9. A point cloud quantization system based on multimodal cues, characterized in that: It includes a text input module, a point cloud input module, a prototype generation module, a point cloud quantization module, a feature fusion module, and a classification and recognition module; The text input module is used to receive text input and extract text features through a text encoder; the point cloud input module is used to receive point cloud data and extract point cloud features through a point cloud encoder; the prototype generation module is used to construct a category prototype based on text features and optimize the prototype using learnable hints; the point cloud quantization module is used to quantize point cloud features to the prototype feature space using Gumbel-Softmax technology. The feature fusion module is used to fuse the quantized point cloud features with the original point cloud features; The classification and recognition module is used to classify point clouds based on fused features, thereby improving the accuracy and generalization ability of point cloud recognition.
10. A computer-readable storage medium, characterized in that: A computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-8.