A multi-modal dynamic fine-tuning method and system for a point cloud large model

Abstract

This invention discloses a multimodal dynamic fine-tuning method and system for large point cloud models. Based on the target large model, the method includes: converting point cloud data into point cloud sequences and 2D depth maps; adding placeholders to text data for subsequent steps; constructing a dynamic multimodal information collector to process the preprocessed 2D depth maps and text, collecting information from the text domain and the 2D image domain; constructing a multimodal cue token generator to generate dynamic cue word parameters suitable for the point cloud encoder of the target large model based on the point cloud sequences and the information collected by the multimodal information collector; embedding the dynamic multimodal information collector and the multimodal cue token generator into the target large model; and training and adjusting the target large model using a dataset and a cross-entropy loss function. This invention achieves precise fine-tuning of the model by dynamically mining the multimodal information of point cloud data, thereby significantly improving the model's performance and generalization ability on various downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of large model technology, specifically to a multimodal dynamic fine-tuning method and system for large point cloud models. Background Technology

[0002] Large point cloud models play a crucial role in downstream tasks of 3D vision detection. However, collecting and labeling high-quality 3D point cloud data is both time-consuming and expensive. Fine-tuning large point cloud models to adapt to downstream tasks faces the problem of insufficient data, which limits the training effect of deep learning models. Although existing fine-tuning methods can improve the performance of models on specific tasks, traditional methods, such as full fine-tuning of large models, are prone to overfitting when data is limited. At the same time, large models have a large number of parameters that need to be updated, consuming a lot of computational resources, which limits their practical application.

[0003] The patent with publication number CN117252986A, entitled "An Efficient Fine-tuning Method and System for Parameters of a 3D Pre-trained Large Model," discloses the following technical features: 3D point cloud data is segmented and encoded to form a point cloud token sequence; 3D features from the downstream task training dataset are used as prior knowledge to construct a point cloud prior library; in the encoder module of the pre-trained model, learnable cue tokens are added before the point cloud token sequence, and a parameterless attention mechanism is used, combined with the prior knowledge in the point cloud prior library, to enhance the learnable cue tokens; the enhanced cue tokens are clustered through a geometric perception adapter, and after local feature interaction through a self-attention mechanism, adjusted tokens are obtained; the adjusted tokens are input into the downstream task head to obtain the predicted output.

[0004] The patent with publication number CN118314566A and titled "Training Method for Point Cloud 3D Detection Model, Point Cloud 3D Detection Method and Device" discloses the following technical features: acquiring point cloud data; inputting the point cloud data into the main branch network and the bypass branch network of a pre-trained model; training the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain a point cloud 3D detection model.

[0005] The main drawbacks of the technical solutions disclosed in the above existing patents can be summarized as follows:

[0006] 1. Existing methods typically require a large amount of task-specific point cloud data to fine-tune the model, which is not only computationally expensive, but may also lead to overfitting when the amount of data is insufficient.

[0007] 2. Insufficient dynamic adjustment capability: Existing technologies lack the ability to make effective dynamic adjustments based on limited data, which limits the adaptability and flexibility of the model to new tasks or data distributions.

[0008] 3. Insufficient utilization of cross-modal knowledge: Point clouds contain cross-modal information. Point clouds can be transformed into 2D images from multiple perspectives through projection, containing rich additional information. However, existing methods have failed to fully utilize this information to improve model performance, missing the opportunity to improve learning and generalization capabilities. Summary of the Invention

[0009] The purpose of this invention is to provide a multimodal dynamic fine-tuning method and system for large point cloud models. Under the premise of consuming only a small amount of computing resources and using limited training data, the method achieves precise fine-tuning of the model by dynamically mining the multimodal information of point cloud data, thereby significantly improving the model's performance and generalization ability on various downstream tasks.

[0010] The technical solution to achieve the purpose of this invention is as follows:

[0011] A multimodal dynamic fine-tuning method for large point cloud models, based on a target large model, is characterized by comprising:

[0012] Step S1, Training data preprocessing: Convert point cloud data into point cloud sequences and 2D depth maps, and add placeholders to text data for use in subsequent steps;

[0013] Step S2: Construct a dynamic multimodal information collector to process the 2D depth map and text obtained from the preprocessing in Step 1, and collect information from the text domain and the 2D image domain.

[0014] Step S3: Construct a multimodal cue token generator. Based on the point cloud sequence in step S1 and the information collected by the multimodal information collector in step S2, generate dynamic cue word parameters suitable for the point cloud encoder of the target large model.

[0015] Step S4: Embed the dynamic multimodal information collector and multimodal cue token generator into the target large model, and use the dataset to train and adjust the target large model through the cross-entropy loss function.

[0016] Furthermore, the point cloud data is converted into a point cloud sequence through the point cloud embedding layer of the target large model.

[0017] Furthermore, the point cloud data is converted into a 2D depth map through the projection module. Specifically, this includes: voxelizing the point cloud, densifying it using local minimum pooling, smoothing its shape using a nonparametric Gaussian kernel, compressing its dimensions, and projecting it from different perspectives to obtain the 2D depth map.

[0018] Furthermore, the placeholder uses 6 characters.

[0019] Furthermore, the dynamic multimodal information collector includes a 2D depth map collection module and a text modal information collection module, wherein:

[0020] The 2D depth map collection module uses a visual encoder with a large-scale pre-trained ViT architecture to extract features from the 2D depth map and convert it into an image feature sequence. This image feature sequence provides information from the 2D modality. The 2D depth map collection module is connected to a multimodal cue token generator and a pre-trained text encoder of the target large model.

[0021] The text modality information collection module uses the text embedding layer of the target large model to obtain word embedding vectors. It then replaces placeholder spaces at the end of the word embedding vectors and embeds learnable cue tokens to obtain a word embedding vector feat with the learnable cue tokens embedded. Text The text modality information collection module is connected to the multimodal cue token generator and the pre-trained image encoder of the target large model, respectively.

[0022] Furthermore, the multimodal cue token generator transforms the learnable cue tokens in the text modality and the image feature sequences in the 2D image modality obtained in step S2 into shapes consistent with the point cloud sequences generated in step S1 using a fully connected layer, and adds the transformation layer to the end of the point cloud sequences to obtain a point cloud sequence that integrates 2D image modality knowledge and text knowledge. In the fine-tuning, only the parameters of the two fully connected layers in the dynamic multimodal information collector and the multimodal cue token generator are updated, while other parameters remain unchanged.

[0023] Furthermore, the cross-entropy loss function is:

[0024]

[0025] Where Cross_entropy is the cross-entropy loss function, and T is the classification label corresponding to the point cloud.

[0026] A multimodal dynamic fine-tuning system for large point cloud models includes:

[0027] The training data preprocessing unit converts point cloud data into point cloud sequences and 2D depth maps, and adds placeholders to text data;

[0028] The dynamic multimodal information collector building unit processes the pre-processed 2D depth map and text to collect information from the text domain and the 2D image domain;

[0029] The multimodal cue token generator building unit generates dynamic cue word parameters suitable for a target large model point cloud encoder based on the step point cloud sequence and information collected by the multimodal information collector.

[0030] The model training fine-tuning unit embeds a dynamic multimodal information collector and a multimodal cue token generator into the target large model. Using the dataset, the target large model is trained and tuned through the cross-entropy loss function.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] (1) This application proposes to integrate knowledge from different modalities (such as text, images and point clouds) through a dynamic multimodal information collector, which significantly improves the model’s ability to understand and represent tasks compared with existing technologies that usually only use single modal information.

[0033] (2) The dynamic multimodal information collector and multimodal cue token generator enable the model to be dynamically adaptable and generate corresponding cue tokens for each data input of different downstream tasks, while the static methods of the existing technology usually cannot be flexibly adjusted.

[0034] (3) The method proposed in this paper does not require modification of the parameters of the pre-trained encoder. Compared with some fine-tuning methods in the prior art, it greatly reduces the need for parameter updates, thereby reducing the consumption of computing resources and the risk of overfitting. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method of the present invention.

[0036] Figure 2 This is an architecture diagram of a multimodal dynamic fine-tuning method proposed in this invention. Detailed Implementation

[0037] This invention proposes a multimodal dynamic fine-tuning method, such as... Figure 1 As shown, it includes the following steps:

[0038] Step S1, Training Data Preprocessing: The original training data for fine-tuning consists of point clouds and their corresponding text labels. For the original point clouds, processing is performed in two modalities: point cloud and 2D image. In the point cloud modality, a point cloud embedding layer is used to convert the point cloud into a point cloud sequence. In the 2D image modality, a projection module is used to project the point cloud into a 2D depth map. For the text labels corresponding to the point clouds, placeholders are added at the end.

[0039] Specifically, in terms of point cloud modality, the input point cloud is embedded using a point cloud embedding layer of the target large model that requires fine-tuning, converting the point cloud into a point cloud sequence. In 2D image modality, the projection module performs the following steps on the input point cloud data: voxelization, local minimum pooling for density enhancement to ensure visual continuity of the projection result, shape smoothing using a non-parametric Gaussian kernel to remove noise, dimensionality compression, and projection from different viewpoints to obtain a 2D depth map. These four steps project the point cloud data into a smooth, visually continuous depth map, reducing the impact of modal differences when projecting the point cloud into the 2D depth map. For the text labels corresponding to the point cloud, a placeholder is added at the end; in this example, we use the 6-character string 'XXXXXX' as the placeholder.

[0040] Step S2, Dynamic Multimodal Information Collection: Use a dynamic multimodal information collector to collect information from the text domain and relevant information from the 2D image domain.

[0041] Specifically, in the text modality, for the text content processed in step S1, after obtaining word embedding vectors through the text embedding layer of the target large model that requires fine-tuning, we embed learnable cue tokens at the end of them. In the image modality, we introduce a 2D domain-trained visual encoder based on the ViT architecture, which processes the 2D depth map obtained from the point cloud projection in step S1 into an image feature sequence. Therefore, in the 2D image modality, additional knowledge matching the point cloud data is dynamically and flexibly provided according to the input of different point clouds.

[0042] Step S3, Multimodal Cue Token Generation: For the information collected by the multimodal information collector, the multimodal cue token generation module is used to generate dynamic cue word parameters suitable for the point cloud encoder.

[0043] Specifically, for the learnable cue tokens in the text modality and the image feature sequences in the image modality in step S2, a fully connected layer is used to transform them into the same shape as the point cloud sequence generated in step S1, and then overlays are added to the end of the point cloud sequence. That is... in, The Prompt token represents the current point cloud. T This represents a learnable cue token in the text modality. It is the image feature sequence corresponding to the current point cloud, f T→P (*) and f I→P (*) represent the corresponding fully connected layers.

[0044] Step S4, Fine-tuning Training: Using the dataset from the downstream task, fine-tune the target large model by adding the dynamic multimodal information collector in S2 and the multimodal cue token generator in step S3 using the cross-entropy loss function. In the fine-tuning, only the parameters of the learnable token in the text encoder part and the two fully connected layers in the multimodal cue token generator are updated, while other parameters remain unchanged.

[0045] Right now

[0046] Among them, feat Text ←f T ([Embed text (T),Prompt T Embed text (T) is the word embedding vector obtained after the original text passes through the target large model's word embedding layer. Prompt T It is a learnable cue token, f T (*) represents the text encoder of the target model;

[0047] Embed point (P i () is the point cloud sequence obtained after the point cloud is embedded into the target large model point cloud. It is the dynamic multimodal cue token obtained in step S3, f P (*) is the point cloud encoder of the target model.

[0048] After fine-tuning, input the corresponding prompts for the point cloud and downstream tasks to perform inference and downstream tasks.

[0049]

[0050] A multimodal dynamic fine-tuning system for large point cloud models includes:

[0051] The training data preprocessing unit converts point cloud data into point cloud sequences and 2D depth maps, and adds placeholders to text data;

[0052] The dynamic multimodal information collector building unit processes the pre-processed 2D depth map and text to collect information from the text domain and the 2D image domain;

[0053] The multimodal cue token generator building unit generates dynamic cue word parameters suitable for a target large model point cloud encoder based on the step point cloud sequence and information collected by the multimodal information collector.

[0054] The model training fine-tuning unit embeds a dynamic multimodal information collector and a multimodal cue token generator into the target large model. Using the dataset, the target large model is trained and tuned through the cross-entropy loss function.

[0055] A computer storage medium storing an executable program, the executable program being executed by a processor to implement the steps of the multimodal dynamic fine-tuning method.

[0056] The multimodal dynamic fine-tuning method of this invention has been experimentally verified and performs excellently in both standard few-shot learning and the more challenging full-class few-shot learning settings, significantly improving the performance of large point cloud models in 3D vision detection tasks, and proving the feasibility and effectiveness of the method in practical applications.

[0057] Table 1 presents a comparison of various point cloud-based large model fine-tuning methods using the standard N-way K-shot few-shot learning setting on the ModelNet40 benchmark dataset. The mean classification accuracy (%) and standard deviation (%) are reported after 10 independent experiments. The number of trainable parameters (millions) indicates the number of trainable parameters (millions) in the model during the fine-tuning or cue-based tuning phase.

[0058] Table 1 Comparison results of various point cloud large model fine-tuning methods

[0059]

[0060]

[0061] Table 2 shows a comparison of different point cloud data detection methods using the K-shot full-class few-shot learning setting on the ModelNet40 and ScanObjectNN benchmark datasets. The method described in this patent (DMMPT) demonstrates superior performance on both datasets.

[0062] Table 2 compares the results of K-shot full-class few-shot learning on the ModelNet40 and ScanObjectNN datasets.

[0063]

[0064] The dynamic multimodal information collector of this invention allows the model to dynamically collect task- and data-related information from different modalities, namely text modality and 2D image modality, and provide it to the 3D point cloud domain. Traditional large point cloud models only possess static knowledge from the point cloud and text domains. The dynamic collection mechanism proposed in this method not only combines rich knowledge from other large models in the 2D domain to supplement additional data-related information, but also adjusts the required knowledge in real time according to the current downstream task, thereby improving the model's adaptability and learning ability to new tasks. Compared with traditional single-modal or static information collection methods, the dynamic multimodal information collector can more effectively utilize cross-modal knowledge, providing richer contextual information for the point cloud encoder and helping to solve the learning problem under data-scarce conditions.

[0065] The multimodal cue token generator of this invention utilizes knowledge obtained from a dynamic multimodal information collector to generate cue tokens that can guide a pre-trained point cloud model to perform specific downstream tasks. This generator is designed to enable the model to more accurately align the knowledge of the pre-trained encoder with the requirements of the current task. By sharing information across different modalities, it enhances the model's understanding of point cloud data, improving learning efficiency and performance.

[0066] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal dynamic fine-tuning method for large point cloud models, based on a target large model, characterized in that, include: Step S1, Training data preprocessing: Convert point cloud data into point cloud sequences and 2D depth maps, and add placeholders to text data; Step S2: Construct a dynamic multimodal information collector to process the 2D depth map and text obtained from the preprocessing in Step 1, and collect information from the text domain and the 2D image domain. Step S3: Construct a multimodal cue token generator. Based on the point cloud sequence in step S1 and the information collected by the multimodal information collector in step S2, generate dynamic cue word parameters suitable for the point cloud encoder of the target large model. Step S4: Embed the dynamic multimodal information collector and multimodal cue token generator into the target large model, and use the dataset to train and adjust the target large model through the cross-entropy loss function; The dynamic multimodal information collector includes a 2D depth map collection module and a text modal information collection module. The 2D depth map collection module uses a visual encoder with a large-scale pre-trained ViT architecture to extract features from the 2D depth map, converting it into an image feature sequence. This module is connected to both a multimodal cue token generator and a pre-trained text encoder of the target large model. The text modal information collection module uses the text embedding layer of the target large model to obtain word embedding vectors. Placeholder spaces are replaced at the end of the word embedding vectors to embed learnable cue tokens, resulting in a word embedding vector *feat* with the learnable cue tokens embedded. Text The text modality information collection module is connected to the multimodal cue token generator and the pre-trained image encoder of the target large model, respectively. The multimodal cue token generator transforms the learnable cue tokens in the text modality and the image feature sequences in the 2D image modality obtained in step S2 into shapes consistent with the point cloud sequences generated in step S1 using a fully connected layer. These shapes are then overlaid onto the end of the point cloud sequences, resulting in a point cloud sequence that integrates 2D image modality knowledge and text knowledge. The cross-entropy loss function is: Where Cross_entropy is the cross-entropy loss function, and T is the classification label corresponding to the point cloud.

2. The multimodal dynamic fine-tuning method for large point cloud models according to claim 1, characterized in that, The point cloud data is converted into a point cloud sequence through the point cloud embedding layer of the target large model.

3. The multimodal dynamic fine-tuning method for large point cloud models according to claim 1, characterized in that, The projection module converts point cloud data into a 2D depth map, specifically by: voxelizing the point cloud, densening it using local minimum pooling, smoothing its shape using a nonparametric Gaussian kernel, compressing its dimensions, and projecting it from different perspectives to obtain the 2D depth map.

4. The multimodal dynamic fine-tuning method for large point cloud models according to claim 1, characterized in that, The placeholder uses 6 characters.

5. A multimodal dynamic fine-tuning system for implementing the multimodal dynamic fine-tuning method according to any one of claims 1-4, characterized in that, include: The training data preprocessing unit converts point cloud data into point cloud sequences and 2D depth maps, and adds placeholders to text data; The dynamic multimodal information collector building unit processes the pre-processed 2D depth map and text to collect information from the text domain and the 2D image domain; The multimodal cue token generator building unit generates dynamic cue word parameters suitable for a target large model point cloud encoder based on the step point cloud sequence and information collected by the multimodal information collector. The model training fine-tuning unit embeds a dynamic multimodal information collector and a multimodal cue token generator into the target large model. Using the dataset, the target large model is trained and tuned through the cross-entropy loss function. The dynamic multimodal information collector includes a 2D depth map collection module and a text modal information collection module. The 2D depth map collection module uses a visual encoder with a large-scale pre-trained ViT architecture to extract features from the 2D depth map, converting it into an image feature sequence. This module is connected to both a multimodal cue token generator and a pre-trained text encoder of the target large model. The text modal information collection module uses the text embedding layer of the target large model to obtain word embedding vectors. Placeholder spaces are replaced at the end of the word embedding vectors to embed learnable cue tokens, resulting in a word embedding vector *feat* with the learnable cue tokens embedded. Text The text modality information collection module is connected to the multimodal cue token generator and the pre-trained image encoder of the target large model, respectively. The multimodal cue token generator transforms the learnable cue tokens in the text modality and the image feature sequences in the 2D image modality obtained in step S2 into shapes consistent with the point cloud sequences generated in step S1 using a fully connected layer. These shapes are then overlaid onto the end of the point cloud sequences, resulting in a point cloud sequence that integrates 2D image modality knowledge and text knowledge. The cross-entropy loss function is: Where Cross_entropy is the cross-entropy loss function, and T is the classification label corresponding to the point cloud.

6. A computer storage medium, characterized in that, The computer storage medium stores an executable program, which is executed by a processor to implement the steps of the multimodal dynamic fine-tuning method according to any one of claims 1-4.

Images