A tuning system and method for point cloud understanding

Abstract

This paper discloses a tuning system for point cloud understanding, comprising a point cloud embedding module, a prompt embedding module, and a pre-trained model. The point cloud embedding module extracts input point cloud features, the prompt embedding module generates prompts online for specified downstream tasks, and the pre-trained model derives output features based on the point cloud features and prompts to tune the task head.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a tuning system and method for point cloud understanding. Background Art

[0002] Currently, 3D general-purpose large models are receiving increasing attention both domestically and internationally, with applications in areas such as autonomous driving and robotics. When using 3D general-purpose large models, a fine-tuning and adaptation algorithm is often required to fine-tune the knowledge of the general model to specific subtasks.

[0003] Currently, there are two mainstream fine-tuning methods: full tuning and adjusting only the task head. Full tuning involves adjusting both the general model and the task head simultaneously. This method has low parameter utilization and often requires a large number of training samples. Adjusting only the task head generally results in lower accuracy. Summary of the Invention

[0004] To address some or all of the problems in the prior art, the present invention provides, on one hand, a tuning system for point cloud understanding, comprising:

[0005] Point cloud embedding module, used to extract input point cloud features;

[0006] A prompt embed module for generating prompts online; and

[0007] A pre-trained model is used to obtain features based on the point cloud features and prompts to adjust the task head.

[0008] Furthermore, the prompt embedding module includes a multi-layer perceptron.

[0009] Furthermore, the pre-training model is based on an attention mechanism and includes at least one transformer layer.

[0010] Furthermore, the pre-training model structure and parameters remain fixed.

[0011] Based on the aforementioned tuning system, the present invention further provides a tuning method for point cloud understanding, comprising:

[0012] Extract point cloud features through point cloud embedding module;

[0013] Generate prompts via the prompt embedding module; and

[0014] The point cloud features and prompts are input into the pre-trained model to adjust the task head.

[0015] Furthermore, generating prompts through the prompt embedding module includes:

[0016] Embed the initialization value input prompt into the module to get the prompt.

[0017] Furthermore, the generation of the initialization value includes:

[0018] Run the pre-trained model on all task-based training datasets to obtain pre-trained features;

[0019] Perform a maximum pooling operation on the pre-trained features of each training dataset in the channel dimension to obtain the maximum value; and

[0020] Calculate the mean of the maximum values corresponding to all training data sets and use it as the initialization value.

[0021] Furthermore, inputting the point cloud features and prompts into a pre-trained model to obtain features includes:

[0022] Inputting the point cloud features and the prompts into the first backbone layer to obtain the first output features;

[0023] Inputting the first output feature and the prompt into the second backbone layer to obtain a second output feature;

[0024] And so on, until the N-1th output feature and the prompt are input into the Nth backbone layer to obtain the final output feature, where N is the number of backbone layers in the pre-trained model.

[0025] Furthermore, the tuning method further includes:

[0026] Computing a loss function to optimize the hint embedding module and / or task head.

[0027] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described above are performed.

[0028] The present invention provides a tuning system and method for point cloud understanding. This system adds a prompt embedding module to existing tuning systems to generate task-specific prompts and pre-train them along with the point cloud. During pre-training, this tuning system and method does not require changing the structure and parameters of the pre-trained model. For different tasks, only different prompts need to be generated and the task header fine-tuned, significantly improving the utilization of the pre-trained model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more detailed description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings depict only typical embodiments of the present invention and are not to be considered as limiting the scope thereof. In the drawings, for clarity, identical or corresponding components will be represented by the same or similar reference numerals.

[0030] Figure 1 A schematic diagram showing the structure of an existing tuning system is shown;

[0031] Figure 2 A schematic diagram illustrating the structure of a tuning system for point cloud understanding according to an embodiment of the present invention; and

[0032] Figure 3 A schematic flow chart illustrating a method for optimizing point cloud understanding according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that the various embodiments can be implemented without one or more of the specific details or with other alternative and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive aspects of the present invention. Similarly, for the purpose of explanation, specific numbers and configurations are set forth in order to provide a comprehensive understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details. In addition, it should be understood that the various embodiments shown in the drawings are illustrative representations and are not necessarily drawn to scale.

[0034] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The appearances of the phrase "in one embodiment" in various places in this specification are not necessarily all referring to the same embodiment.

[0035] It should be noted that the embodiments of the present invention describe the method steps in a specific order, but this is only for the purpose of illustrating the specific embodiment and does not limit the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0036] The current fine-tuning methods for point cloud understanding mainly include full tuning and subset tuning. Full tuning means that the complete pre-trained model parameters need to be updated, that is, it is necessary to copy and store the entire pre-trained model parameters for each downstream task. As the model scale grows, storing and fine-tuning the pre-trained model for each downstream task will greatly increase the burden on the client or user end, and the parameter utilization efficiency is not high. Subset tuning only updates the task header or model bias, and its accuracy is low. In response to the above problems, the present invention considers applying the prompt tuning method to point cloud understanding. Among them, the prompt tuning method is currently mostly used in natural language processing (NLP) and 2D image recognition tasks. It is to add a series of marks to the input text and / or two-dimensional image, and the marks can be called prompts. The prompt tuning method only updates the prompts and task headers when training a specified task, without the need for each task amplitude and training model, so it can effectively improve the parameter utilization efficiency. However, unlike NLP and 2D images, 3D point cloud data is unstructured data that describes structures. Therefore, it is difficult to pre-design structural templates. This makes it impossible to obtain prompt initialization values through structural templates, and it is also difficult to pre-design and store prompt libraries. Based on this, the present invention provides a tuning system and method for point cloud understanding. Based on the existing tuning system, it adds a prompt embedded module to generate a task-specific meta-structure online and use it as a prompt for point cloud tasks. The meta-structure can be used as a condition for fine-tuning a specific task, such as maximizing the probability of object classification labels for a given point cloud.

[0037] The solution of the present invention will be further described below in conjunction with the accompanying drawings of the embodiments.

[0038] Figure 1 The schematic diagram of the existing tuning system is shown in FIG. In the figure, the gray modules represent the modules or parameters that need to be adjusted. Figure 1 As shown, the existing tuning system includes a point cloud embedding module (point embed) 101 and a pre-trained model 102. The pre-trained model 102 uses a Transformer encoder structure based on the attention mechanism. For different downstream tasks, the existing tuning system needs to update the entire model parameters, including the pre-trained model and the task head (Head), and adjust the point cloud embedding module (point embed) and the pre-trained model to solve different tasks.

[0039] Figure 2FIG. 1 is a schematic diagram showing the structure of a tuning system for point cloud understanding according to an embodiment of the present invention. In the figure, the gray modules represent the modules or parameters that need to be adjusted. Figure 2 As shown, a tuning system for point cloud understanding includes a point cloud embedding module 201, a prompt embedding module 202, and a pre-trained model 203. It can be seen that compared with existing tuning systems, the tuning system for point cloud understanding provided by the present invention adds a prompt embedding module 202, which enables it to generate task-based prompts according to specified downstream tasks, thereby generating task heads for different tasks without updating or adjusting the pre-trained model, greatly improving the efficiency of using the parameters of the pre-trained model.

[0040] The point cloud embedding module 201 is used to extract the features E1, E2...E of the input point cloud. N The feature extraction structure and / or method used by the point cloud embedding module 201 is basically consistent with the prior art and will not be described in detail here.

[0041] The prompt embed module 202 is used to generate prompts P online for a specified downstream task. M Since a multi-layer perceptron (MLP) can learn a complex mathematical function to represent a continuous manifold structure, in one embodiment of the present invention, the prompt embedding module 202 includes at least one layer of MLP, through which meta-structures can be generated online to serve as prompts for downstream tasks. In one embodiment of the present invention, the prompt embedding module 202 includes a three-layer MLP.

[0042] The features E1, E2...E N and prompt P M The input is input into the pre-training model 203 to obtain output features, which are further input into the task head of the downstream task to complete the tuning. In one embodiment of the present invention, the pre-training model 203 adopts a neural network structure Transformer based on the attention mechanism, which includes at least one backbone layer Transformer layer, the prompt P MIt is input into each Transformer layer, so that the final output features contain task-based information. In actual application, the tuning system provided by the present invention only needs to generate different initialization parameters according to different tasks, input them into the prompt embed module 202 to obtain task-based prompts, and then update the task head. It does not need to adjust or modify the structure and parameters of the pre-trained model, or in other words, it freezes the pre-trained model. The pre-trained model can be used for different downstream tasks, effectively improving the parameter usage efficiency of the pre-trained model.

[0043] Based on the tuning system described above, Figure 3 A flow chart of a method for optimizing point cloud understanding according to an embodiment of the present invention is shown. Figure 3 As shown, a tuning method for point cloud understanding includes:

[0044] First, in step 301, point cloud features are extracted. The point cloud features of the input point cloud are extracted by the point cloud embedding module 201, and H0 = [E1, E2 ..., E N ];

[0045] Next, in step 302, prompts are generated. Based on a specific downstream task, the prompt embedded module 202 generates a task-based prompt P M :

[0046] P M =Embed(F ini ),

[0047] Wherein, Embed() refers to the prompt embedding module 202, which, in one embodiment of the present invention, specifically includes a multi-layer MLP, such as a three-layer MLP; and

[0048] F ini is the initialization feature of the prompt embedding module 202. In the prompt tuning method, the initialization of the prompt has a great influence on the accuracy of the prompt generated by the prompt embedding module 202, which in turn affects the accuracy of the downstream tasks of the point cloud. Therefore, how to reasonably obtain the initialization features associated with the task is crucial. In one embodiment of the present invention, the initialization features are generated based on the pre-trained features of the specified task. Specifically, the pre-trained model TF is run on all task-based training datasets TD to obtain the pre-trained features F p ∈R L×C , and then obtain the initialization feature F by averaging the maximum pool pre-training features on the channel dimension ini ∈R L :

[0049]

[0050] Where L is the length of the prompt, C is the channel size of the pre-trained feature, and N is the number of training samples in the training dataset. refers to the maximum pooling operation in the channel dimension, which can obtain unique features; and

[0051] Finally, in step 303, the task head is adjusted. The point cloud features and prompts are input into the pre-trained model to adjust the task head. In one embodiment of the present invention, the pre-trained model TF uses a Transformer model based on the attention mechanism as the backbone network, which includes K layers of backbone layers Transformer layer TF i , i=1,2,…,K, adjusting the task head through the Transformer model includes:

[0052] The point cloud feature H0=[E1,E2…,E N ] and prompt P M Input to the first layer

[0053] Dry the layer to get the first output feature H1:

[0054] H1=TF1(H0,P M );

[0055] The first output feature H1 is combined with the prompt P M Input to the second backbone layer to obtain the second output feature H2:

[0056] H2=TF2(H1,P M );

[0057] And so on, until the K-1th output feature is combined with the prompt P M Input to the Kth backbone layer to obtain the final output feature H K :

[0058] H i =TF i (H i-1 , P M ), i=1,2,…,K;

[0059] The final output feature H K Feed it into the task head to adjust the task head.

[0060] In one embodiment of the present invention, a loss function is also calculated to optimize the prompt embedding module and / or task head. Using the aforementioned tuning system and method, only the prompt embedding module and task head need to be trained for a specific downstream task, without the need to store the pre-trained model multiple times.

[0061] After testing, the tuning method and system provided by the present invention have performance in various tasks such as object classification, few-shot learning, and part segmentation that is basically equivalent to or even better than the existing full-tuning tuning method. At the same time, the tuning method and system provided by the present invention are significantly more efficient in parameter usage than the existing tuning methods, and the parameters stored and updated only account for approximately 3% of the total model parameters.

[0062] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not limitation. It will be apparent to those skilled in the relevant art that various combinations, modifications, and variations may be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely in accordance with the appended claims and their equivalents.

Claims

1. A tuning system for point cloud understanding, characterized in that include: a point cloud embedding module configured to extract input point cloud features; a prompt embedding module, which is configured to generate prompts online for a specified downstream task; as well as A pre-trained model is configured to input the point cloud features and prompts into the pre-trained model to obtain output features to adjust the task head of the downstream task, specifically including: inputting the point cloud features and prompts into the first backbone layer to obtain the first output feature, and then inputting the first output feature and prompts into the second backbone layer to obtain the second output feature, and so on, until the K-1th output feature and prompt are input into the Kth backbone layer to obtain the final output feature, wherein K is the number of backbone layers in the pre-trained model.

2. The tuning system according to claim 1, wherein: The prompt embedding module includes a multi-layer perceptron.

3. The tuning system according to claim 1, wherein: The pre-training model is based on an attention mechanism and includes at least one backbone layer.

4. The tuning system according to claim 1, wherein: The pre-training model structure and parameters remain fixed.

5. A tuning method for point cloud understanding, characterized in that The method is implemented based on the tuning system according to any one of claims 1 to 4, and comprises the steps of: Extract point cloud features through point cloud embedding module; Generate prompts for the specified downstream task through the prompt embedding module; as well as The point cloud features and prompts are input into a pre-trained model to obtain output features, and the output features are fed into the task head of the downstream task.

6. The tuning method according to claim 5, wherein: Generating prompts through the prompt embed module includes: The initialization value is input into the prompt embedding module and the prompt is generated through the multi-layer perceptron.

7. The tuning method according to claim 6, wherein: The generation of the initialization value includes: Run the pre-trained model on all task-based training datasets to obtain pre-trained features; Perform a maximum pooling operation on the pre-trained features of each training dataset in the channel dimension to obtain the maximum value; and Calculate the mean of the maximum values corresponding to all training data sets and use it as the initialization value.

8. The tuning method according to claim 6, wherein: Also includes: A loss function is calculated to optimize the hint embedding module and / or the task head.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the method according to any one of claims 5 to 8.

Images