Multi-modal dynamic fine tuning method and system for large point cloud model

Abstract

The invention discloses a multi-modal dynamic fine tuning method and system for a point cloud large model, and the method is based on a target large model, and comprises the steps: converting point cloud data into a point cloud sequence and a 2D depth map, and adding placeholders to text data for subsequent steps; a dynamic multi-modal information collector is constructed, the 2D depth map and the text obtained through preprocessing are processed, and information from the text field and the 2D image field is collected; constructing a multi-modal prompt token generator, and generating dynamic prompt word parameters of the point cloud encoder suitable for the target large model based on the point cloud sequence and the information collected by the multi-modal information collector; and embedding the dynamic multi-modal information collector and the multi-modal prompt token generator into the target large model, and training and adjusting the target large model through a cross entropy loss function by using the data set. By dynamically mining the multi-modal information of the point cloud data, accurate fine adjustment of the model is realized, so that the performance and generalization ability of the model on various downstream tasks are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and more specifically, to a multimodal dynamic fine-tuning method and system for point cloud large models. Background Art

[0002] Large point cloud models play a key role in downstream tasks of 3D visual inspection, but since collecting and annotating high-quality 3D point cloud data is time-consuming and expensive, fine-tuning large point cloud models to adapt to downstream tasks faces the problem of insufficient data, which limits the training effect of deep learning models. Although existing fine-tuning methods can improve the performance of models on specific tasks, using traditional methods, such as fully fine-tuning large models, is prone to the risk of overfitting when data is limited. At the same time, since large models have a large number of parameters that need to be updated, they consume a lot of computing resources, which limits their practical applications.

[0003] The patent with publication number CN117252986A and name "A method and system for efficient fine-tuning of parameters of a three-dimensional pre-trained large model" discloses the following technical features: dividing the three-dimensional point cloud data into blocks and encoding them to form a point cloud token sequence; building a point cloud prior library using the 3D features in the downstream task training data set as prior knowledge; and in the encoder module of the pre-trained model, before adding the learnable prompt token to the point cloud token sequence, a parameter-free attention mechanism is used, and the learnable prompt token is enhanced in combination with the prior knowledge in the point cloud prior library; the enhanced prompt tokens are clustered through a geometry-aware adapter, and local feature interactions are performed through a self-attention mechanism to obtain adjusted tokens; the adjusted tokens are input into the downstream task head to obtain a predicted output.

[0004] The patent with publication number CN118314566A and name of training method of point cloud 3D detection model, point cloud 3D detection method and device discloses the technical features: obtaining point cloud data; inputting the point cloud data into the main branch network and bypass branch network of the pre-trained model, and training the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain a point cloud 3D detection model.

[0005] The main disadvantages of the technical solutions disclosed in the above existing patents can be summarized as follows:

[0006] 1. Existing methods usually require a large amount of task-specific point cloud data to fine-tune the model, which is not only computationally expensive but may also lead to overfitting of the model when the amount of data is insufficient.

[0007] 2. Insufficient dynamic adjustment capabilities: Existing technologies lack the ability to make effective dynamic adjustments based on limited data, which limits the adaptability and flexibility of the model on new tasks or data distributions.

[0008] 3. Insufficient use of cross-modal knowledge: Point clouds contain cross-modal information. Point clouds can be converted into 2D images from multiple perspectives through projection, which contain rich additional information. However, existing methods fail to make full use of this information to improve model performance, missing the opportunity to improve learning and generalization capabilities. Summary of the invention

[0009] The purpose of the present invention is to provide a multimodal dynamic fine-tuning method and system for large point cloud models. Under the premise of consuming only a small amount of computing resources and using limited training data, the multimodal information of point cloud data is dynamically mined to achieve accurate fine-tuning of the model, thereby significantly improving the performance and generalization ability of the model in various downstream tasks.

[0010] The technical solution to achieve the purpose of the present invention is:

[0011] A multimodal dynamic fine-tuning method for a point cloud large model, based on a target large model, is characterized by comprising:

[0012] Step S1, training data preprocessing: converting point cloud data into point cloud sequence and 2D depth map, and adding placeholders to text data for use in subsequent steps;

[0013] Step S2, constructing a dynamic multimodal information collector, processing the 2D depth map and text obtained by the preprocessing in step 1, and collecting information from the text field and the 2D image field;

[0014] Step S3, constructing a multimodal prompt token generator, and generating dynamic prompt word parameters of a point cloud encoder suitable for the target large model based on the point cloud sequence of step S1 and the information collected by the multimodal information collector of step S2;

[0015] Step S4, embed the dynamic multimodal information collector and the multimodal prompt token generator into the target large model, use the data set, and train and adjust the target large model through the cross entropy loss function.

[0016] Furthermore, the point cloud data is converted into a point cloud sequence through the point cloud embedding layer of the target large model.

[0017] Furthermore, the point cloud data is converted into a 2D depth map through a projection module, which specifically includes: voxelizing the point cloud, densifying it using a local minimum pooling operation, smoothing the shape using a non-parametric Gaussian kernel, and finally compressing the dimension and projecting it from different perspectives to obtain a 2D depth map.

[0018] Furthermore, the placeholder uses 6 characters.

[0019] Furthermore, the dynamic multimodal information collector includes a 2D depth map collection module and a text modality information collection module, wherein:

[0020] The 2D depth map collection module uses a large-scale pre-trained ViT architecture visual encoder to extract features from the 2D depth map and convert the 2D depth map into an image feature sequence, which provides information from the 2D modality. The 2D depth map collection module is connected to the multimodal prompt token generator and the pre-trained text encoder of the target large model respectively;

[0021] The text modality information collection module uses the text embedding layer of the target large model to obtain a word embedding vector, and replaces the placeholder space at the end of the word embedding vector to embed a learnable prompt token to obtain a word embedding vector feat embedded with the learnable prompt token Text , the text modality information collection module is respectively connected to the multimodal prompt token generator and the pre-trained image encoder of the target large model.

[0022] Furthermore, the multimodal prompt token generator uses a fully connected layer to convert the learnable prompt token in the text mode and the image feature sequence in the 2D image mode obtained in step S2 into a shape consistent with the point cloud sequence generated in step S1, and adds them to the end of the point cloud sequence to obtain a point cloud sequence that integrates 2D image modality knowledge and text knowledge. In fine-tuning, only the parameters of two fully-connected layers in the dynamic multimodal information collector and the multimodal prompt tokenizer are updated, and the other parameters remain unchanged.

[0023] Furthermore, the cross entropy loss function is:

[0024]

[0025] Among them, Cross_entropy is the cross entropy loss function, and T is the classification label corresponding to the point cloud.

[0026] A multi-modal dynamic fine-tuning system for large point cloud models, comprising:

[0027] The training data preprocessing unit converts the point cloud data into a point cloud sequence and a 2D depth map, and adds a placeholder to the text data;

[0028] A dynamic multimodal information collector construction unit processes the preprocessed 2D depth map and text to collect information from the text field and the 2D image field;

[0029] A multimodal prompt token generator construction unit generates dynamic prompt word parameters of a point cloud encoder suitable for a target large model based on the step point cloud sequence and the information collected by the multimodal information collector;

[0030] The model training fine-tuning unit embeds the dynamic multimodal information collector and the multimodal prompt token generator into the target large model, uses the dataset, and trains and adjusts the target large model through the cross entropy loss function.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) This application proposal uses a dynamic multimodal information collector to integrate knowledge from different modalities (such as text, images, and point clouds). Compared with the existing methods that usually only use single modality information, this significantly improves the model's ability to understand and represent tasks;

[0033] (2) The dynamic multimodal information collector and multimodal prompt token generator make the model dynamically adaptable and can generate corresponding prompt tokens for each data input of different downstream tasks, while the static methods in the prior art are usually not flexible enough;

[0034] (3) The proposed method does not require modification of the parameters of the pre-trained encoder. Compared with some fine-tuning methods in the prior art, it significantly reduces the need for parameter updates, thereby reducing the consumption of computing resources and the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flow chart of the method of the present invention.

[0036] Figure 2 This is an architecture diagram of a multi-modal dynamic fine-tuning method proposed in the present invention. DETAILED DESCRIPTION

[0037] The present invention proposes a multi-modal dynamic fine-tuning method, such as Figure 1 As shown, the following steps are included:

[0038] Step S1, training data preprocessing: The original training data for fine-tuning is: point cloud and its corresponding text label. For the original point cloud, the original point cloud is processed from two modalities: point cloud and 2D image. In the point cloud modality, the point cloud is processed using the point cloud embedding layer to convert the point cloud into a point cloud sequence. In the 2D image modality, the point cloud is projected into a 2D depth map through the projection module. For the text label corresponding to the point cloud, a placeholder is added at the end.

[0039] Specifically, in the point cloud modality, the input point cloud is converted into a point cloud sequence using the point cloud embedding layer of the target large model that needs to be fine-tuned. In the 2D image modality, the projection module takes the following actions on the input point cloud data: voxelize the point cloud, densify it using the local minimum pooling operation to ensure the visual continuity of the projection result, then use the non-parametric Gaussian kernel for shape smoothing, remove noise, and finally compress the dimension, project from different perspectives, and obtain a 2D depth map. After these four steps, the point cloud data is projected into a smooth visually continuous depth map, reducing the impact of modality differences when the point cloud is projected into the 2D depth map. For the text label corresponding to the point cloud, a placeholder is added at the end. In the example, we use the 6-bit character 'XXXXXX' as a placeholder.

[0040] Step S2, dynamic multimodal information collection: using a dynamic multimodal information collector to collect information from the text field and related information from the 2D image field.

[0041] Specifically, in the text modality, for the text content processed in step S1, we first pass the word embedding vector through the text embedding layer of the target large model that needs to be fine-tuned, and then embed a learnable prompt token at the end. In the image modality, we introduce a large-scale trained ViT-based visual encoder in the 2D field to process the 2D depth map obtained by projecting the point cloud in step S1 into an image feature sequence. Therefore, in the 2D image modality, additional knowledge matching the point cloud data can be dynamically and flexibly provided with different point cloud inputs.

[0042] Step S3, multimodal prompt token generation: for the information collected by the multimodal information collector, use the multimodal prompt token generation module to generate dynamic prompt word parameters suitable for the point cloud encoder

[0043] Specifically, for the learnable hint tokens in the text mode and the image feature sequence in the image mode in step S2, a fully connected layer is used to transform them into the same shape as the point cloud sequence generated in step S1, and the overlay is added to the end of the point cloud sequence. in, Indicates the prompt token corresponding to the current point cloud, Prompt T represents a learnable cue token in text mode, is the image feature sequence corresponding to the current point cloud, f T→P (*) and f I→P (*) are the corresponding fully connected layers.

[0044] Step S4, fine-tuning training: Using the dataset of the downstream task, fine-tune the target large model with the dynamic multimodal information collector in S2 and the multimodal prompt token generator in step S3 through the cross entropy loss function. In the fine-tuning, only the parameters of the learnable tokens in the text encoder part and the two fully connected layers in the multimodal prompt token generator are updated, and other parameters remain unchanged.

[0045] Right now

[0046] Among them, feat. Text ←f T ([Embed text (T), Prompt T ]), Embed text (T) is the word embedding vector obtained after the original text passes through the word embedding layer of the target model, Prompt T is a learnable cue token, f T (*) is the text encoder of the target model;

[0047] Embed point (P i ) is the point cloud sequence obtained after the point cloud passes through the target large model point cloud embedding layer, is the dynamic multimodal prompt word token obtained in step S3, f P (*) is the point cloud encoder of the target model.

[0048] After fine-tuning is completed, input the prompt words corresponding to the point cloud and downstream tasks to perform reasoning and downstream tasks.

[0049]

[0050] A multi-modal dynamic fine-tuning system for large point cloud models, comprising:

[0051] The training data preprocessing unit converts the point cloud data into a point cloud sequence and a 2D depth map, and adds a placeholder to the text data;

[0052] A dynamic multimodal information collector construction unit processes the preprocessed 2D depth map and text to collect information from the text field and the 2D image field;

[0053] A multimodal prompt token generator construction unit generates dynamic prompt word parameters of a point cloud encoder suitable for a target large model based on the step point cloud sequence and the information collected by the multimodal information collector;

[0054] The model training fine-tuning unit embeds the dynamic multimodal information collector and the multimodal prompt token generator into the target large model, uses the dataset, and trains and adjusts the target large model through the cross entropy loss function.

[0055] A computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the multi-modal dynamic fine-tuning method.

[0056] The multimodal dynamic fine-tuning method of the present invention has been experimentally verified to perform well in both standard few-shot learning and more challenging all-category few-shot learning settings, significantly improving the performance of large point cloud models in 3D vision detection tasks, demonstrating the feasibility and effectiveness of the method in practical applications.

[0057] Table 1 shows the comparison between various point cloud large model fine-tuning methods using the standard N-way K-shot few-shot learning setting on the ModelNet40 benchmark dataset. The average classification accuracy (%) and standard deviation (%) of 10 independent experiments are reported. Trainable Parameters (millions) indicates the number of trainable parameters (millions) in the model during the fine-tuning or hint adjustment stage.

[0058] Table 1 Comparison results between various point cloud large model fine-tuning methods

[0059]

[0060]

[0061] Table 2 shows the comparison between different point cloud data detection methods using K-shot all-category few-shot learning settings on the ModelNet40 and ScanObjectNN benchmark datasets. The method in this patent (DMMPT) shows excellent performance on both datasets.

[0062] Table 2 Comparison results of K-shot full-category few-shot learning on ModelNet40 and ScanObjectNN datasets

[0063]

[0064] The dynamic multimodal information collector of the present invention includes a mechanism that allows the model to dynamically collect task and data related information from different modalities, namely text modality and 2D image modality, and provide it to the 3D point cloud field. Traditional large point cloud models only have static knowledge from the point cloud and text fields, and the dynamic collection mechanism proposed by this method can not only combine the rich knowledge from other large models in the 2D field and supplement additional data-related information, but also adjust the required knowledge in real time according to the current downstream tasks, thereby improving the model's adaptability and learning ability for new tasks. Compared with traditional unimodal or static information collection methods, the dynamic multimodal information collector can more effectively utilize cross-modal knowledge, provide richer contextual information for the point cloud encoder, and help solve learning problems in data-scarce situations.

[0065] The multimodal hint token generator of the present invention uses the knowledge obtained from the dynamic multimodal information collector to generate hint tokens that can guide the pre-trained point cloud model to perform specific downstream tasks. The design of this generator enables the model to more accurately align the knowledge of the pre-trained encoder with the requirements of the current task, and by sharing information between different modalities, the model's understanding of point cloud data is enhanced, improving learning efficiency and performance.

[0066] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A multimodal dynamic fine-tuning method for a point cloud large model, based on a target large model, characterized in that: include: Step S1, training data preprocessing: converting point cloud data into point cloud sequence and 2D depth map, and adding placeholders to text data; Step S2, constructing a dynamic multimodal information collector, processing the 2D depth map and text obtained by the preprocessing in step 1, and collecting information from the text field and the 2D image field; Step S3, constructing a multimodal prompt token generator, and generating dynamic prompt word parameters of a point cloud encoder suitable for the target large model based on the point cloud sequence of step S1 and the information collected by the multimodal information collector of step S2; Step S4, embed the dynamic multimodal information collector and the multimodal prompt token generator into the target large model, use the data set, and train and adjust the target large model through the cross entropy loss function.

2. A multi-modal dynamic fine-tuning method for a large point cloud model according to claim 1, characterized in that: The point cloud data is converted into a point cloud sequence through the point cloud embedding layer of the target large model.

3. The multi-modal dynamic fine-tuning method for a large point cloud model according to claim 1, characterized in that: The point cloud data is converted into a 2D depth map through the projection module, which includes: voxelizing the point cloud, densifying it using the local minimum pooling operation, smoothing the shape using a non-parametric Gaussian kernel, and finally compressing the dimension and projecting it from different perspectives to obtain a 2D depth map.

4. The multimodal dynamic fine-tuning method for a large point cloud model according to claim 1, characterized in that: The placeholder uses 6 characters.

5. The multi-modal dynamic fine-tuning method for a large point cloud model according to claim 1, characterized in that: The dynamic multimodal information collector includes a 2D depth map collection module and a text modal information collection module. The 2D depth map collection module uses a large-scale pre-trained ViT architecture visual encoder to extract features from the 2D depth map and convert the 2D depth map into an image feature sequence. The 2D depth map collection module is respectively connected to a multimodal prompt token generator and a pre-trained text encoder of a target large model; the text modal information collection module uses a text embedding layer of the target large model to obtain a word embedding vector, replaces a placeholder space at the end of the word embedding vector, embeds a learnable prompt token, and obtains a word embedding vector feat embedded with the learnable prompt token Text , the text modality information collection module is respectively connected to the multimodal prompt token generator and the pre-trained image encoder of the target large model.

6. A multi-modal dynamic fine-tuning method for a large point cloud model according to claim 5, characterized in that: The multimodal prompt token generator uses a fully connected layer to convert the learnable prompt token in the text mode and the image feature sequence in the 2D image mode obtained in step S2 into a shape consistent with the point cloud sequence generated in step S1, and overwrites and adds them to the end of the point cloud sequence to obtain a point cloud sequence that integrates 2D image modality knowledge and text knowledge.

7. A multi-modal dynamic fine-tuning method for a large point cloud model according to claim 6, characterized in that: The cross entropy loss function is: Among them, Cross_entropy is the cross entropy loss function, and T is the classification label corresponding to the point cloud.

8. A multi-modal dynamic fine-tuning system for implementing the multi-modal dynamic fine-tuning method according to any one of claims 1 to 7, characterized in that: include: The training data preprocessing unit converts the point cloud data into a point cloud sequence and a 2D depth map, and adds a placeholder to the text data; A dynamic multimodal information collector construction unit processes the preprocessed 2D depth map and text to collect information from the text field and the 2D image field; A multimodal prompt token generator construction unit generates dynamic prompt word parameters of a point cloud encoder suitable for a target large model based on the step point cloud sequence and the information collected by the multimodal information collector; The model training fine-tuning unit embeds the dynamic multimodal information collector and the multimodal prompt token generator into the target large model, uses the dataset, and trains and adjusts the target large model through the cross entropy loss function.

9. A computer storage medium, characterized in that The computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the multimodal dynamic fine-tuning method according to any one of claims 1 to 7.

Images