3D object classification continuous learning method and system based on parameter level mask adapter

Abstract

The invention discloses a 3D object classification continuous learning method and system based on a parameter level mask adapter, and the method comprises the steps: obtaining to-be-detected 3D point cloud data, and extracting the 2D features, original 3D features and text features of the 3D point cloud data; fusing the 2D features and the 3D features through a parameter-level mask adapter to obtain global features; performing comparative learning on the global features and the text features to obtain a 3D object classification result; wherein when the parameter level mask adapter continuously learns a plurality of 3D object classification tasks, the most relevant parameters of the learned tasks are frozen, and the most relevant parameters of the learned tasks are not updated any more in subsequent tasks. According to the method, the prior knowledge of the CLIP model is adapted to a 3D object classification task through multi-modal feature fusion, and continuous learning based on parameter isolation is realized by using a mask of a parameter level, so that disastrous forgetting is effectively reduced, and the classification performance is improved. The invention further provides a new model continuous learning ability evaluation index, and a more comprehensive evaluation framework is provided for 3D object classification continuous learning.

Description

Technical Field

[0001] The present invention relates to the field of 3D continuous learning technology, and in particular to a 3D object classification continuous learning method and system based on a parameter-level mask adapter. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] 3D object classification plays a vital role in many fields such as autonomous driving, robot navigation, and virtual reality. In these application scenarios, the model needs to be able to continuously adapt to new environments and new categories. However, existing continuous learning methods mainly focus on 2D tasks, and there are great technical challenges when directly applying these methods to 3D tasks. For example, there are domain differences between 2D data and 3D data, and 3D point cloud data has the characteristics of disorder and unstructuredness, making it difficult to extract meaningful features. In addition, the number of samples in actual application scenarios is usually limited, which can easily lead to overfitting of the model during training. At the same time, there is often significant semantic transfer between 3D tasks, especially when previous task data is not available, the model is prone to catastrophic forgetting.

[0004] In recent years, the success of pre-trained visual language models on cross-modal datasets has provided a potential solution to these problems. Such models significantly improve the quality of 3D feature learning by transferring 2D knowledge. In addition, rich prior knowledge, as a broad knowledge base, helps bridge the semantic gap between different tasks, thereby reducing the risk of catastrophic forgetting.

[0005] The prior art discloses methods such as improving the performance of point cloud classification by integrating image and depth features, using 2D pre-trained models to guide mask autoencoders for feature learning, unifying the representation of text, images and point clouds, and using pre-trained models to enhance few-sample incremental learning. However, these methods mainly focus on how to use pre-trained models to improve the feature extraction capabilities of 3D tasks, but do not address the challenge of continuous learning of 3D tasks in practical applications. Moreover, since these pre-trained models are usually designed for a wide range of tasks, when directly applied to specific 3D tasks, the performance is often difficult to achieve optimal. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a continuous learning method and system for 3D object classification based on a parameter-level mask adapter. Through multimodal feature fusion, the prior knowledge of the CLIP (Contrastive Language–Image Pre-training) model is adapted to the 3D object classification task, and parameter-level masks are used to realize continuous learning based on parameter isolation, which effectively reduces catastrophic forgetting and improves classification performance. At the same time, two new evaluation indicators are proposed to evaluate the continuous learning ability of the entire model.

[0007] In some embodiments, the following technical solutions are adopted:

[0008] A parameter-level mask adapter-based continuous learning method for 3D object classification, comprising:

[0009] Obtain the 3D point cloud data to be tested, and extract the 2D features, original 3D features and text features of the 3D point cloud data respectively;

[0010] The 2D features and the 3D features are fused by a parameter-level mask adapter to obtain a global feature;

[0011] The global features are compared with the text features to obtain a 3D object classification result;

[0012] When the parameter-level mask adapter continuously learns multiple 3D object classification tasks, the most relevant parameters of the learned tasks are frozen, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks.

[0013] As an optional solution, obtain the 2D features and text features of the 3D point cloud data, specifically:

[0014] The acquired 3D point cloud data to be tested is projected into two sets of depth images at different distances at multiple angles and rendered into a set of RGB images at the same time. The 2D encoder in the CLIP model is used to extract and aggregate features from the depth image and RGB image to obtain 2D features.

[0015] The text encoder in the CLIP model is used to extract text features.

[0016] As an optional solution, the original 3D features of the 3D point cloud data are obtained, specifically: the 3D point cloud data is encoded by a 3D encoder to obtain the original 3D features.

[0017] As an optional solution, the parameter-level mask adapter includes a 3D part, a 2D part and a global part; the 3D part is used to process the original 3D features to obtain features f aligned with the 2D feature dimension p ; The 2D part is used to combine the 2D features with the features fp Perform feature fusion to obtain image feature f I ; The global part is used to perform feature f p and feature f I Perform addition and averaging to obtain the fused global feature f g .

[0018] As an optional solution, the global features are compared with the text features to obtain the 3D object classification results, specifically:

[0019] The global features and text features are processed by the principal component projection method to obtain the global features that remove redundant information. and text features

[0020] Global features to remove redundant information and text features The cosine similarity calculation is performed, and the prediction probability of each category is obtained by normalizing the calculation results. The category with the highest probability is selected as the 3D object classification result.

[0021] As an alternative, the overall training loss for the 3D object classification task is:

[0022] L t =L cls,t +αL cont,t ;

[0023]

[0024] Among them, L cls,t is the classification loss of task t, L cont is the contrastive learning loss of task t, α represents the weight of contrastive learning loss; is the number of samples for this task, N Aug is the augmentation number, L InfoNCE is the information noise contrast estimation loss, and the enhancement function f Aug Generate the camera direction, is the i-th sample from f Aug (x i,t ) calculates the j-th augmented global feature, x i,t is the i-th 3D point cloud sample input, represents the positive samples of the corresponding category, while all other categories As a negative sample, y i,t is the i-th sample label, L ce is the cross entropy loss, p i,t represents the predicted probability of the i-th sample in task t.

[0025] As an alternative, the parameter-level mask adapter freezes the most relevant parameters of the learned tasks while continuously learning multiple 3D object classification tasks, specifically:

[0026] Assuming that the entire network is parameterized by weight θ as f(x;θ), each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as the weight score s;

[0027] When training each task, first randomly initialize all parameters of the parameter-level mask adapter, including the linear layer weight θ and its corresponding weight score s; the top c% of weights with the highest weight scores represent the parameters most relevant to the current task, and their weight masks are assigned a value of 1, and the remaining weight masks are assigned a value of 0; when training the current task, weights with a mask value of 1 participate in the training;

[0028] During back-propagation, the masked union of previously learned tasks is obtained through Boolean OR operations, indicating the most relevant weights of all previously learned tasks, and their corresponding gradients are set to zero to ensure that the learned weights are not destroyed during the current task training.

[0029] As an optional solution, the continuous learning ability of the entire neural network model is evaluated through two evaluation indicators: network focus drift and macro forgetting.

[0030] The network focus drift degree is specifically:

[0031]

[0032] The macro forgetting degree is specifically:

[0033]

[0034] Among them, T is the total number of tasks for network training, MAcc t is the macro accuracy of task t, and the calculation formula is: K t is the number of categories in task t, Acc i is the accuracy of category i; NCAcc is the accuracy of novel categories, and the calculation formula is: The number of samples predicted correctly for category i, is the number of samples for this task.

[0035] In other embodiments, the following technical solutions are adopted:

[0036] A parameter-level mask adapter-based continuous learning system for 3D object classification, comprising:

[0037] A feature extraction module is used to obtain the 3D point cloud data to be tested and extract the 2D features, original 3D features and text features of the 3D point cloud data respectively;

[0038] A feature fusion module, used for fusing the 2D features and 3D features to obtain a global feature through a parameter-level mask adapter;

[0039] A contrastive learning module, used for contrastive learning of the global features and text features to obtain a 3D object classification result;

[0040] When the parameter-level mask adapter continuously learns multiple 3D object classification tasks, the most relevant parameters of the learned tasks are frozen, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks.

[0041] In other embodiments, the following technical solutions are adopted:

[0042] A terminal device includes a processor and a memory, wherein the processor is used to implement instructions; the memory is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the above-mentioned parameter-level mask adapter-based 3D object classification continuous learning method.

[0043] In other embodiments, the following technical solutions are adopted:

[0044] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned parameter-level mask adapter-based 3D object classification continuous learning method.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The present invention introduces a visual language pre-training model to enable the network to acquire a large amount of prior knowledge, thereby improving the model's feature learning ability, acting as a unified knowledge base in multi-task continuous learning, and alleviating the catastrophic forgetting phenomenon in 3D multi-task continuous learning.

[0047] The present invention adapts the pre-trained model to the 3D task field through a parameter-level mask adapter, obtains global features by fusing 2D features and 3D features, and enriches the information of point cloud samples by combining the multimodal features of 3D point clouds and 2D images; then, the global features and text features are compared and learned to achieve the classification of 3D objects, thereby improving the classification performance and generalization ability of the model.

[0048] (2) The present invention introduces a parameter-level mask mechanism, which allows the network to effectively select the weights most relevant to the current task, avoiding the transfer of redundant knowledge in the old knowledge that has low relevance to the new task. This allows the network to retain the performance of the old tasks while flexibly and efficiently learning new tasks, thereby continuously learning 3D tasks and ensuring that the model can maintain a high classification accuracy in different tasks and adapt to complex actual scene requirements.

[0049] (3) This paper proposes a new evaluation index for the model's continuous learning ability, providing a more comprehensive evaluation framework for continuous learning of 3D object classification.

[0050] Other features and advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flow chart of a continuous learning method for 3D object classification based on a parameter-level mask adapter in an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of parameter isolation process of parameter-level mask adapter in an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of experimental results of multi-task continuous learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0056] 3D object classification plays a vital role in many fields such as autonomous driving, robot navigation, and virtual reality. In these application scenarios, the model needs to be able to continuously adapt to new environments and new categories. However, existing continuous learning methods mainly focus on 2D tasks. Due to the domain differences between 2D data and 3D data, there are great technical challenges when directly applying these methods to 3D tasks. In recent years, the success of pre-trained visual language models on cross-modal datasets has provided a potential solution for achieving continuous learning for 3D tasks. However, most scholars currently focus more on how to use pre-trained models to improve the feature extraction capabilities of 3D tasks, but have not addressed the challenges of continuous learning in practical applications. Moreover, since these pre-trained models are usually designed for a wide range of tasks, when directly applied to specific 3D tasks, the performance is often difficult to achieve optimal.

[0057] Based on this, the present invention introduces a visual language pre-training model, combines the multimodal features of 3D point cloud, 2D image and text prompts, enriches the information of point cloud samples, and improves the classification performance and generalization ability of the model. The parameter-level mask mechanism is introduced to allow the network to effectively select the weights most relevant to the current task, avoid transferring redundant knowledge in the old knowledge that is less relevant to the new task, and allow the network to retain the performance of the old tasks while learning new tasks flexibly and efficiently, thereby continuously learning 3D tasks, ensuring that the model can maintain a high classification accuracy in different tasks and adapt to complex actual scene requirements.

[0058] Embodiment 1

[0059] In one or more embodiments, a continuous learning method for 3D object classification based on a parameter-level mask adapter is disclosed, combined with Figure 1 , specifically including the following process:

[0060] S101: Acquire 3D point cloud data to be measured, and extract 2D features, original 3D features, and text features of the 3D point cloud data respectively.

[0061] In this embodiment, the 3D point cloud sample (x i,t ,y i,t ) is projected from multiple angles into two sets of depth maps at different distances. At the same time, the 3D model is rendered from each angle to obtain a set of RGB images. The 2D encoder in the visual language pre-training model (CLIP model) is used to extract and aggregate features from the depth image and RGB image to obtain the 2D feature F 2D .

[0062] At the same time, the 3D point cloud sample (x i,t ,y i,t ) Obtain the original 3D feature F through the 3D encoder encoding 3D .

[0063] The 3D point cloud sample (x i,t ,y i,t ) is modeled as a text prompt, and the text encoder is used to extract text features F t .

[0064] As a specific example, this embodiment uses the pre-trained CLIP's ViT-B / 32 as the visual language pre-training model, and uses the pre-trained dynamic graph edge convolutional network (DGCNN) as the 3D encoder.

[0065] S102: fusing 2D features and 3D features through a parameter-level mask adapter to obtain a global feature.

[0066] In this embodiment, in order to avoid losing key information in the dimensionality reduction process, in the 3D part of the parameter-level mask adapter, F 3D After repeated operations and simple feature processing, the 2D feature F 2D Dimensionally aligned features f p , in the 2D part of the parameter-level mask adapter, the feature f p With feature F 2D Perform feature fusion to obtain image feature f I , in the global part of the parameter-level mask adapter, for feature f p and feature f I Add and average to get the fused global feature f g By processing and fusing the features of 3D point clouds and 2D images, the information of point cloud samples is enriched, and multimodal features can be efficiently extracted and learned.

[0067] In this embodiment, the parameter-level mask adapter includes a 3D part, a 2D part and a global part; wherein the 3D part is used to process the original 3D features to obtain features f aligned with the 2D feature dimension. p ; The 2D part is used to combine 2D features with feature f p Perform feature fusion to obtain image feature f I ; The global part is used for feature f p and feature f I Perform addition and averaging to obtain the fused global feature f g .

[0068] In this embodiment, the specific structure of the parameter-level mask adapter includes a 3D part, a 2D part, and a global part; wherein the 3D part contains two linear layers, each of which is followed by batch normalization, followed by a simple repetitive operation to align with the 2D features. The 2D part uses principal component analysis to fuse the input features, first through batch normalization and Dropout layer processing, then flattened into a one-dimensional vector, then through a linear layer for linear transformation, then through batch normalization and ReLU activation function, and finally through the second linear layer to output the global features. The global part performs simple addition and averaging operations on the input 3D and 2D features.

[0069] It should be noted that the linear layers involved in the parameter-level mask adapter are weighted and masked based on the existing structure (given in the following steps), and structures such as batch normalization and Dropout layers, batch normalization and ReLU activation functions can all be implemented using the existing structure.

[0070] In this implementation, the visual language pre-training model CLIP, the process of projecting 3D point cloud samples into two sets of depth maps with different distances, the 3D encoder and the parameter-level mask adapter are combined to form the entire neural network model.

[0071] S103: Perform comparative learning on the global features and the text features to obtain a 3D object classification result.

[0072] In this embodiment, in order to align the feature space of the image and the text, the global feature f g With the text feature F t Perform contrastive learning, process global features and text features through principal component projection method, and obtain global features that remove redundant information and text features The logit of class k (the original output without normalization) is calculated based on these processed features using the cosine similarity:

[0073]

[0074] Finally, the logits of all categories are normalized using the softmax function to obtain the predicted probability of each category, and the category with the highest probability is taken as the 3D object classification result.

[0075] In this embodiment, in order to improve the feature learning ability of the entire neural network model, this embodiment adopts the InfoNCE loss L infoNCE As contrastive learning loss, it is calculated as follows:

[0076]

[0077] Among them, the enhancement function f Aug Generate the camera direction, is the i-th sample from f Aug (x i,t ) calculates the j-th augmented global feature, x i,t is the i-th 3D point cloud sample input; N Aug is the augmented number, represents the positive samples of the corresponding category, while all other categories As negative samples, is the number of samples for this task, L InfoNCE is the information-noise contrast estimation loss.

[0078] The classification loss L for task t cls,t It is calculated as follows:

[0079]

[0080] Among them, L ce is the cross entropy loss, p i,t represents the predicted probability of the i-th sample in task t, y i,t is the label of the ith sample.

[0081] The overall training loss L for task t t It is calculated as follows:

[0082] L t =L cls,t +αL cont,t (4)

[0083] Here, α represents the weight of contrastive learning loss, and in this embodiment, α=1.

[0084] Assume D = {D1, D2, ..., D t ,...,D T} represents a series of 3D object classification tasks, where the tth task D t Contains t training samples, x i,t is the input sample, y i,t belongs to the label set Y t The corresponding label of . Assume that for t≠t′, Y t With Y t″ There is no intersection between them. After completing the training of the tth task, the entire neural network model must classify the test sets of all previous tasks {D1,D2,...,D t D1 is a benchmark task with a large-scale training dataset, while the incremental task D t The number of samples is small, that is, for t>1, n t <<n1.

[0085] In this embodiment, the entire neural network model continuously learns multiple 3D object classification tasks, and the most relevant parameters of the learned tasks need to be frozen through the parameter-level mask adapter to retain the knowledge of the old tasks and alleviate catastrophic forgetting. At the same time, it provides training possibilities for subsequent tasks to the greatest extent, allowing the use of relevant parts of the old knowledge to assist the learning of new knowledge, and balancing the stability and plasticity of the network.

[0086] In this embodiment, it is assumed that the entire neural network is parameterized by weight θ and represented as f(x; θ). The entire neural network is a model determined by the parameter θ. Through training, θ will be adjusted to optimize the performance of the model. The output of the model depends not only on the input x, but also on the parameter θ.

[0087] In order to protect the prior knowledge of the visual language pre-training model from being destroyed, the parameters of the visual language pre-training model used are kept frozen; each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as the weight score s, and the higher the score, the more important it is to the current task.

[0088] In this embodiment, the linear layer is a network layer involved in training in the neural network. The input is linearly transformed through a set of trainable weight matrices and bias terms, and a new vector is output; the input of the indicator function is the score of the scoring layer, and the output is a binary mask of 0 or 1. The linear layer score is input into the indicator function, and the output obtained at the top c% position with the highest score is 1, which means that the parameters with a mask of 1 are more important to the current task, and the rest are 0; the mask is not only used in the current task to indicate the optimal parameter subset of the current task, but also used to set the gradient of the parameters with a mask of 1 to zero during subsequent task training to protect the important parameters of the old task from being changed.

[0089] When training each task, we first randomly initialize all parameters of the parameter-level mask adapter, including the linear layer weights θ and their corresponding weight scores s; the top c% of weights with the highest scores represent the most relevant parameters for the current task, and are indicated by the function The weight mask assigned to this position is 1, and the rest of the weight masks are set to 0;

[0090] During training, each task will generate a mask corresponding to the current task, indicating the most relevant parameters for the current task. The weights with the current task mask of 1 will participate in the training and loss function calculations normally. The weight scores can be updated synchronously as a trainable parameter layer to optimize the evaluation ability of the weight scores.

[0091] The optimal weighted subset for task t is given by θ t =θ⊙m t Determine, where ⊙ represents the element-wise product, m t is a binary mask.

[0092] During the back-propagation process, the union of all learned old task masks is obtained through Boolean OR operation As a cumulative binary mask, ∨ represents the OR operation in Boolean operations, such as mask m1 = (0, 1, 1, 0), mask m2 = (1, 0, 1, 0), and the two are ORed to obtain the union M2 = (1, 1, 1, 0). Among them, the weights with a mask of 1 in the union indicate the most relevant parameters of the previous task. When training the current task, the gradient needs to be zeroed and not updated to protect these old task parameters that have been learned, and prevent the destruction of the old task knowledge that has been learned during subsequent task training.

[0093] Specifically, when training task t, the weight and mask m * t The optimization process is determined by the following formula:

[0094]

[0095] When the subsequent task arrives, the accumulated mask M t-1 To protect the old task weights, the weight score s is updated synchronously to guide the new task-specific mask. The update process is as follows:

[0096]

[0097] where η represents the learning rate of the optimizer. This process allows the network to retain knowledge learned from previous tasks while maintaining flexibility, thus providing intuitive continuous learning capabilities for 3D tasks.

[0098] In order to evaluate the continuous learning ability of the entire neural network model, this embodiment proposes two new evaluation indicators, namely, network focus drift and macro forgetting, specifically:

[0099] Network focus drift σ distract It is used to measure the degree of focus drift of the network when continuously learning multiple tasks. It is calculated in a similar way to variance. The higher the value, the greater the drift of network focus when training multiple tasks, and the more difficult it is for the network to maintain stable performance. The specific calculation formula is as follows:

[0100]

[0101] Among them, T is the total number of tasks for network training, MAcc t is the macro accuracy of task t, and its calculation formula is K t is the number of categories in task t, Acc i is the accuracy of category i, NCAcc is the accuracy of novel categories, and its calculation formula is The number of samples predicted correctly for category i, is the number of samples for this task.

[0102] Macro forgetting degree Δ Macro It is used to evaluate the macro forgetting situation during the entire training process. It evaluates the performance change of each incremental task relative to the previous task. The larger the value, the more likely the neural network model is to experience catastrophic forgetting when training multiple tasks. The specific formula is as follows:

[0103]

[0104] Using the above two model continuous learning ability evaluation indicators can provide more comprehensive and reliable evaluation results for the 3D object classification continuous learning task.

[0105] Reference Figure 2 ,The design of continuous learning using parameter-level masks is as follows:

[0106] When training for a benchmark task, the c% of weights with the highest scores indicate the optimal parameter subset for the benchmark task, whose task-specific masks are set to 1, and the masks of the remaining weights are set to 0. Weights with masks of 1 can participate in training, and the weight scores can be optimized during training to more effectively select the parameters most relevant to the benchmark task.

[0107] When training for incremental task 1, the adapter evaluates the performance of all weights in the linear layer. The c% weights with the highest scores set the incremental task 1 specific mask value to 1 and are selected as the optimal parameter subset for this task. At this time, the cumulative mask is the benchmark task specific mask, and the weight gradients with a cumulative mask of 1 are set to zero to retain the knowledge of the benchmark task.

[0108] When training for incremental task 2, the adapter evaluates the performance of all weights in the linear layer and selects the c% weights with the highest scores, setting their incremental task 2 specific masks to 1 as the optimal parameter subset for incremental task 2. At this time, the cumulative mask contains the specific mask of the baseline task and the specific mask of incremental task 1. The gradients of weights with a cumulative mask of 1 are set to zero to retain the knowledge of the two old tasks. The training of subsequent tasks follows the same process to continue selecting and freezing the optimal parameters.

[0109] During training, the network uses the ADAM weight decay optimizer, the batch size is set to 32, and the learning rate is set to 1×10 -3 , weight decay is set to 1×10 -4 .L InfoNCE The temperature coefficient τ of is set to 0.1 (a lower temperature coefficient makes the model focus more on high similarity samples, while a higher temperature coefficient makes the similarity distribution curve smoother). The training of the baseline task and each incremental task lasts for 10 and 20 rounds, respectively.

[0110] like Figure 3 The figure shows the learning results of the implementation case of the 3D multi-task continuous learning solution proposed in this embodiment. Figure 3 It can be seen that the 3D continuous learning method based on the parameter-level mask adapter proposed in this embodiment has a significant advantage over the method without a parameter-level mask adapter in that the macro accuracy fluctuates less with the arrival of tasks, and the model is more stable, from the basic task (Base Task) to the incremental tasks (Incre-T1, Incre-T2, Incre-T3, Incre-T4, Incre-T5, Incre-T6).

[0111] Embodiment 2

[0112] In one or more embodiments, a parameter-level mask adapter-based 3D object classification continuous learning system is disclosed, comprising:

[0113] A feature extraction module is used to obtain the 3D point cloud data to be tested and extract the 2D features, original 3D features and text features of the 3D point cloud data respectively;

[0114] A feature fusion module, used for fusing the 2D features and 3D features to obtain a global feature through a parameter-level mask adapter;

[0115] A contrastive learning module, used for contrastive learning of the global features and text features to obtain a 3D object classification result;

[0116] When the parameter-level mask adapter continuously learns multiple 3D object classification tasks, the most relevant parameters of the learned tasks are frozen, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks.

[0117] The specific implementation of the above modules is the same as that in Example 1 and will not be described in detail.

[0118] Embodiment 3

[0119] In one or more embodiments, a terminal device is disclosed, which includes a processor and a memory, the processor being used to implement instructions; the memory being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the 3D object classification continuous learning method based on a parameter-level mask adapter described in Example 1.

[0120] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0122] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0123] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A continuous learning method for 3D object classification based on parameter-level mask adapter, characterized in that include: Obtain the 3D point cloud data to be tested, and extract the 2D features, original 3D features and text features of the 3D point cloud data respectively; The 2D features and the 3D features are fused by a parameter-level mask adapter to obtain a global feature; The global features are compared with the text features to obtain a 3D object classification result; When the parameter-level mask adapter continuously learns multiple 3D object classification tasks, the most relevant parameters of the learned tasks are frozen, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks.

2. A method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: Get the 2D features and text features of 3D point cloud data, specifically: The acquired 3D point cloud data to be tested is projected into two sets of depth images at different distances at multiple angles and rendered into a set of RGB images at the same time. The 2D encoder in the CLIP model is used to extract and aggregate features from the depth image and RGB image to obtain 2D features. The text encoder in the CLIP model is used to extract text features.

3. The method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: The original 3D features of the 3D point cloud data are obtained, specifically: the 3D point cloud data is encoded by a 3D encoder to obtain the original 3D features.

4. The method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: The parameter-level mask adapter includes a 3D part, a 2D part and a global part; the 3D part is used to process the original 3D features to obtain features f aligned with the 2D feature dimension. p ; The 2D part is used to combine the 2D features with the features f p Perform feature fusion to obtain image feature f I ; The global part is used for feature f p and feature f I Perform addition and averaging to obtain the fused global feature f g .

5. The method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: The global features are compared with the text features to obtain the 3D object classification results, which are as follows: The global features and text features are processed by the principal component projection method to obtain the global features that remove redundant information. and text features Global features to remove redundant information and text features The cosine similarity calculation is performed, and the prediction probability of each category is obtained by normalizing the calculation results. The category with the highest probability is selected as the 3D object classification result.

6. The method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: The overall training loss for the 3D object classification task is: L t =L cls,t +αL cont,t ; Among them, L cls,t is the classification loss of task t, L cont is the contrastive learning loss of task t, α represents the weight of contrastive learning loss; is the number of samples for this task, N Aug is the augmentation number, L InfoNCE is the information noise contrast estimation loss, and the enhancement function f Aug Generate the camera direction, is the i-th sample from f Aug (x i,t ) calculates the j-th augmented global feature, x i,t is the i-th 3D point cloud sample input, represents the positive samples of the corresponding category, while all other categories As a negative sample, y i,t is the i-th sample label, L ce is the cross entropy loss, p i,t represents the predicted probability of the i-th sample in task t.

7. The method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: The parameter-level mask adapter freezes the most relevant parameters of the learned tasks while continuously learning multiple 3D object classification tasks, specifically: Assuming that the entire network is parameterized by weight θ as f(x;θ), each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as the weight score s; When training each task, first randomly initialize all parameters of the parameter-level mask adapter, including the linear layer weight θ and its corresponding weight score s; the top c% of weights with the highest weight scores represent the parameters most relevant to the current task, and their weight masks are assigned a value of 1, and the remaining weight masks are assigned a value of 0; when training the current task, weights with a mask value of 1 participate in the training; During back-propagation, the masked union of previously learned tasks is obtained through Boolean OR operations, indicating the most relevant weights of all previously learned tasks, and their corresponding gradients are set to zero to ensure that the learned weights are not destroyed during the current task training.

8. The method for continuous learning of 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that: Also includes: The continuous learning ability of the entire neural network model is evaluated through two evaluation indicators: network focus drift and macro forgetting. The network focus drift degree is specifically: The macro forgetting degree is specifically: Among them, T is the total number of tasks for network training, MAcc t is the macro accuracy of task t, and the calculation formula is: K t is the number of categories in task t, Acc i is the accuracy of category i; NCAcc is the accuracy of novel categories, and the calculation formula is: Predict the number of correct samples for category i; is the number of samples for this task.

9. A continuous learning system for 3D object classification based on parameter-level mask adapter, characterized in that include: A feature extraction module is used to obtain the 3D point cloud data to be tested and extract the 2D features, original 3D features and text features of the 3D point cloud data respectively; A feature fusion module, used for fusing the 2D features and 3D features through a parameter-level mask adapter to obtain a global feature; A contrastive learning module, used for contrastive learning of the global features and text features to obtain a 3D object classification result; When the parameter-level mask adapter continuously learns multiple 3D object classification tasks, the most relevant parameters of the learned tasks are frozen, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks.

10. A terminal device, comprising a processor and a memory, wherein the processor is used to implement instructions; and the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the continuous learning method for 3D object classification based on a parameter-level mask adapter as described in any one of claims 1-7.

Images