3D object classification continual learning method and system based on parameter-level mask adapter

Abstract

The application discloses a 3D object classification continuous learning method and system based on a parameter-level mask adapter, comprising: obtaining 3D point cloud data to be measured, and extracting 2D features, original 3D features and text features of the 3D point cloud data respectively; fusing the 2D features and the 3D features through a parameter-level mask adapter to obtain global features; and performing comparative learning on the global features and the text features to obtain a 3D object classification result; wherein the parameter-level mask adapter freezes the most relevant parameters of a learned task when continuously learning multiple 3D object classification tasks, and the most relevant parameters of the learned task are no longer updated in subsequent tasks. The application adapts the prior knowledge of a CLIP model to a 3D object classification task through multi-modal feature fusion, realizes continuous learning based on parameter isolation by using a parameter-level mask, effectively reduces catastrophic forgetting, and improves classification performance. The application also proposes a new model continuous learning capability evaluation index, and provides a more comprehensive evaluation framework for 3D object classification continuous learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D continual learning, and in particular to a 3D object classification continual learning method and system based on a parameter-level mask adapter. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] 3D object classification plays a crucial role in many fields such as autonomous driving, robot navigation, and virtual reality. In these application scenarios, models need to be able to continuously adapt to new environments and new classes. However, existing continual learning methods mainly focus on 2D tasks, and directly applying these methods to 3D tasks faces significant technical challenges, such as domain differences between 2D data and 3D data, unordered and unstructured characteristics of 3D point cloud data, making it difficult to extract meaningful features. And the number of samples in actual application scenarios is usually limited, which can easily lead to overfitting of the model during training. At the same time, there are significant semantic shifts between 3D tasks, especially when previous task data is not available, the model is prone to catastrophic forgetting.

[0004] In recent years, the success of pre-trained visual language models on cross-modal datasets provides a potential solution to these problems. Such models significantly improve the quality of 3D feature learning through the transfer of 2D knowledge. In addition, rich prior knowledge as a broad knowledge base helps to bridge the semantic gap between different tasks, thereby reducing the risk of catastrophic forgetting.

[0005] Existing technologies disclose methods such as integrating image and depth features to improve point cloud classification performance, using 2D pre-trained models to guide mask autoencoders for feature learning, unifying text, image, and point cloud representations, and using pre-trained models to enhance few-shot incremental learning. However, these methods mainly focus on how to use pre-trained models to improve the feature extraction capability of 3D tasks, but do not solve the challenge of continual learning for 3D tasks in actual applications. And because these pre-trained models are usually designed for broad tasks, when directly applied to specific 3D tasks, the performance is often difficult to achieve optimality. SUMMARY

[0006] To solve the above problems, the application provides a 3D object classification lifelong learning method and system based on a parameter-level mask adapter, prior knowledge of a CLIP (Contrastive Language-Image Pre-training) model is adapted to a 3D object classification task through multi-modal feature fusion, lifelong learning based on parameter isolation is realized by using a parameter-level mask, catastrophic forgetting is effectively reduced, and classification performance is improved; and two new evaluation indexes are provided for evaluating lifelong learning capability of the whole model.

[0007] In some embodiments, the following technical solutions are adopted:

[0008] A 3D object classification lifelong learning method based on a parameter-level mask adapter, comprising:

[0009] Obtaining 3D point cloud data to be tested, 2D features, original 3D features and text features of the 3D point cloud data are extracted respectively;

[0010] Fusing the 2D features and the 3D features through a parameter-level mask adapter to obtain global features;

[0011] Comparatively learning the global features and the text features to obtain a 3D object classification result;

[0012] The parameter-level mask adapter freezes the most relevant parameters of a learned task when performing lifelong learning on multiple 3D object classification tasks, and the most relevant parameters of the learned task are not updated in subsequent tasks.

[0013] As an optional solution, the 2D features and the text features of the 3D point cloud data are obtained, and specifically:

[0014] The 3D point cloud data to be tested is projected into two groups of depth images with different distances and simultaneously rendered into a group of RGB images; a 2D encoder in a CLIP model is used to extract and aggregate features of the depth images and the RGB images to obtain 2D features;

[0015] A text encoder in the CLIP model is used to extract text features.

[0016] As an optional solution, the original 3D features of the 3D point cloud data are obtained, and specifically, the 3D point cloud data is encoded by a 3D encoder to obtain original 3D features.

[0017] As an optional solution, the parameter-level mask adapter comprises a 3D part, a 2D part and a global part; the 3D part is used to process the original 3D features to obtain features f p that are aligned with the dimensions of the 2D features; the 2D part is used to align the 2D features with the features fp The feature fusion is performed to obtain an image feature f I The global part is used to add and average process the features f p and f I to obtain a fused global feature f g .

[0018] As an optional solution, the global feature is compared with the text feature to obtain a 3D object classification result, specifically:

[0019] The global feature and the text feature are processed by a principal component projection method to obtain a global feature f and a text feature f

[0020] The global feature f and the text feature f are subjected to cosine similarity calculation, and the prediction probability of each category is obtained by normalizing the calculation result, and the category with the highest probability is selected as the 3D object classification result.

[0021] As an optional solution, the total training loss of the 3D object classification task is:

[0022] L t =L cls,t +αL cont,t ;

[0023]

[0024] Wherein, L cls,t is the classification loss of task t, L cont is the contrastive learning loss of task t, and α represents the weight of the contrastive learning loss; is the number of samples of the task, N Aug is the augmented number, L InfoNCE is the information noise contrast estimation loss, and the enhancement function f Aug generates a camera direction, is the global feature of the i-th sample calculated by f Aug (x i,t ) for the j-th augmentation, x i,t is the i-th input 3D point cloud sample, represents the positive sample of the corresponding category, and all other categories are as negative samples, y i,t is the label of the i-th sample, L ce is the cross-entropy loss, and p i,t represents the prediction probability of the i-th sample in task t.

[0025] As an optional solution, the parameter-level mask adapter freezes the most relevant parameters of the learned task when continuously learning multiple 3D object classification tasks, specifically:

[0026] Assuming that the entire network is parameterized by weights θ and represented as f(x; θ), each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as weight scores s;

[0027] During training for each task, all parameters of the parameter-level mask adapter are first randomly initialized, including linear layer weights θ and their corresponding weight scores s; the top c% of weights with the highest weight scores represent the most relevant parameters for the current task, and their weight masks are assigned a value of 1, while the remaining weight masks are assigned a value of 0; during training for the current task, weights with a mask value of 1 participate in training;

[0028] During backpropagation, the mask union set of previously learned tasks is obtained through Boolean OR operation, indicating the most relevant weights of all previously learned tasks, and their corresponding gradients are set to zero to ensure that the learned weights are not damaged during training for the current task.

[0029] As an optional solution, the continuous learning ability of the entire neural network model is evaluated by two evaluation indicators: network focus drift and macro-forgetting;

[0030] The network focus drift is specifically:

[0031]

[0032] The macro-forgetting is specifically:

[0033]

[0034] Where T is the total number of network training tasks, MAcc t is the macro accuracy of task t, and the calculation formula is: K t is the number of classes within task t, Acc i is the accuracy of class i; NCAcc is the novel class accuracy, and the calculation formula is: is the number of samples predicted correctly for class i, is the number of samples for the task.

[0035] In some other embodiments, the following technical solutions are adopted:

[0036] A 3D object classification continuous learning system based on a parameter-level mask adapter, comprising:

[0037] The feature extraction module is configured to obtain 3D point cloud data to be tested, and extract 2D features, original 3D features and text features of the 3D point cloud data respectively.

[0038] The feature fusion module is configured to fuse the 2D features and the 3D features through a parameter-level mask adapter to obtain global features.

[0039] The contrast learning module is configured to perform contrast learning on the global features and the text features to obtain a 3D object classification result.

[0040] In the parameter-level mask adapter, the most relevant parameters of a learned task are frozen during continuous learning of multiple 3D object classification tasks, and the most relevant parameters of the learned task are no longer updated in subsequent tasks.

[0041] In some other embodiments, the following technical solutions are adopted:

[0042] A terminal device includes a processor and a memory, the processor is configured to implement instructions, and the memory is configured to store a plurality of instructions, the instructions are adapted to be loaded and executed by the processor to implement the 3D object classification continuous learning method based on the parameter-level mask adapter.

[0043] In some other embodiments, the following technical solutions are adopted:

[0044] A computer-readable storage medium stores a plurality of instructions, the instructions are adapted to be loaded and executed by the processor of the terminal device to implement the 3D object classification continuous learning method based on the parameter-level mask adapter.

[0045] Compared with the prior art, the beneficial effects of the present application are:

[0046] (1) The present application introduces a visual language pre-training model, so that the network obtains a large amount of prior knowledge and improves the feature learning ability of the model, and serves as a unified knowledge base in multi-task continuous learning, thereby alleviating the catastrophic forgetting phenomenon in 3D multi-task continuous learning.

[0047] The present application adapts the pre-training model to the 3D task field through the parameter-level mask adapter, fuses the 2D features and the 3D features to obtain global features, combines the multi-modal features of the 3D point cloud and the 2D image, and enriches the information of the point cloud sample; then the global features and the text features are contrastively learned to realize the classification of the 3D object, and the classification performance and the generalization ability of the model are improved.

[0048] (2) The application introduces a parameter-level mask mechanism, allowing the network to effectively select the weights most relevant to the current task, avoiding the transmission of redundant knowledge in old knowledge with low relevance to new tasks, allowing the network to retain old task performance while flexibly and efficiently learning new tasks, thereby continuously learning 3D tasks, ensuring that the model can maintain high classification accuracy in different tasks and adapt to complex actual scene requirements.

[0049] (3) The application proposes a new model continuous learning capability evaluation index, providing a more comprehensive evaluation framework for 3D object classification continuous learning.

[0050] Other features and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A 3D object classification continuous learning method based on a parameter-level mask adapter in an embodiment of the application is shown in the flowchart.

[0052] Figure 2 A parameter-level mask adapter parameter isolation process schematic diagram in an embodiment of the application is shown in the flowchart.

[0053] Figure 3 An experimental result schematic diagram of multi-task continuous learning in an embodiment of the application is shown in the flowchart. DETAILED DESCRIPTION

[0054] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0055] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0056] 3D object classification plays a crucial role in various fields such as autonomous driving, robot navigation, and virtual reality. In these application scenarios, models need to be able to adapt to new environments and new categories continuously. However, existing continual learning methods mainly focus on 2D tasks, and due to the domain difference between 2D data and 3D data, there are significant technical challenges in directly applying these methods to 3D tasks. In recent years, the success of pre-trained visual language models on cross-modal datasets provides a potential solution for implementing continual learning for 3D tasks. However, most researchers focus on how to use pre-trained models to improve the feature extraction capability of 3D tasks, but they do not address the challenges of continual learning in practical applications. Moreover, since these pre-trained models are usually designed for general tasks, their performance is often suboptimal when directly applied to specific 3D tasks.

[0057] Based on this, the present application enriches the information of point cloud samples and improves the classification performance and generalization ability of the model by introducing a visual language pre-trained model and combining the multi-modal features of 3D point cloud, 2D image, and text prompt. The introduction of the parameter-level mask mechanism allows the network to effectively select the most relevant weights for the current task, avoiding the transfer of redundant knowledge with low relevance to the new task from old knowledge, allowing the network to retain the performance of old tasks while flexibly and efficiently learning new tasks, thereby continuously learning 3D tasks and ensuring that the model maintains high classification accuracy in different tasks and adapts to complex real-world scenario requirements.

[0058] Embodiment one

[0059] In one or more embodiments, a 3D object classification continual learning method based on a parameter-level mask adapter is disclosed, which combines Figure 1 , and specifically includes the following processes:

[0060] S101: Obtain the 3D point cloud data to be tested, and extract the 2D features, original 3D features, and text features of the 3D point cloud data respectively.

[0061] In this embodiment, the 3D point cloud sample (x i,t ,y i,t ) is projected into two groups of depth maps with different distances from multiple angles, and a 3D model is rendered from each angle to obtain a group of RGB images. The 2D encoder in the visual language pre-trained model (CLIP model) is used to extract and aggregate the features of the depth images and RGB images, obtaining the 2D features F 2D .

[0062] At the same time, the 3D point cloud sample (x i,t ,y i,t ) is encoded by a 3D encoder to obtain the original 3D features F 3D .

[0063] The category label of the 3D point cloud sample (x i,t ,y i,t ) is patterned into a text prompt, and a text encoder is used to extract text features F t .

[0064] As a specific example, the embodiment uses the pre-trained CLIP ViT-B / 32 as the visual language pre-training model, and uses the pre-trained dynamic graph edge convolution network (DGCNN) as the 3D encoder.

[0065] S102: Fuse the 2D features and the 3D features through the parameter-level mask adapter to obtain global features.

[0066] In the embodiment, in order to avoid losing key information in the dimension reduction process, in the 3D part of the parameter-level mask adapter, F 3D is processed through repeated operations and simple feature processing to obtain the feature f p which is dimensionally aligned with the 2D feature F 2D . p In the 2D part of the parameter-level mask adapter, the feature f p is fused with the feature F 2D to obtain the image feature f I . p In the global part of the parameter-level mask adapter, the feature f p and the feature f I are added and averaged to obtain the fused global feature f g . Through the feature processing and fusion of the 3D point cloud and the 2D image, the information of the point cloud sample is enriched, and the multi-modal features can be efficiently extracted and learned.

[0067] In the embodiment, the parameter-level mask adapter includes a 3D part, a 2D part, and a global part; wherein the 3D part is used to process the original 3D feature to obtain a feature f p which is dimensionally aligned with the 2D feature; the 2D part is used to fuse the 2D feature with the feature f p to obtain an image feature f I ; and the global part is used to add and average the feature f p and the feature f I to obtain the fused global feature f g .

[0068] In this embodiment, the specific structure of the parameter-level mask adapter includes a 3D part, a 2D part and a global part; wherein the 3D part contains two linear layers, each of which is followed by batch normalization processing, and then a simple repetition operation is performed to align with the 2D feature. The 2D part fuses the input features through principal component analysis, first passes through the batch normalization and Dropout layer processing, then flattens into a one-dimensional vector, then passes through a linear layer for linear transformation, and then passes through batch normalization and ReLU activation function, and finally passes through the second linear layer to output the global feature. The global part is a simple addition and mean operation of the input 3D and 2D features.

[0069] It should be noted that the linear layer involved in the parameter-level mask adapter has undergone weight fraction and mask processing (which will be given later) based on the existing structure, and the batch normalization and Dropout layer, batch normalization and ReLU activation function and other structures can be realized by using the existing structure.

[0070] In this embodiment, the visual language pre-training model CLIP, the process of projecting the 3D point cloud sample into two groups of different distance depth maps, the 3D encoder and the parameter-level mask adapter are combined to form the entire neural network model.

[0071] S103: The global feature and the text feature are compared and learned to obtain the 3D object classification result.

[0072] In this embodiment, in order to align the feature spaces of the image and the text, the global feature f g is compared and learned with the text feature F t , and the global feature f and the text feature F are processed through principal component projection to obtain the global feature f and the text feature F

[0073] with redundant information removed.

[0074] Finally, the logits of all classes are normalized using the softmax function to obtain the prediction probability of each class, and the class with the highest probability is taken as the 3D object classification result.

[0075] In this embodiment, in order to improve the feature learning ability of the entire neural network model, the InfoNCE loss L infoNCE is used as the contrastive learning loss, which is calculated as follows:

[0076]

[0077] Among them, the enhancement function f Aug Generate camera direction, is the i-th sample from f Aug (x i,t ) calculates the j-th augmented global feature, x i,t is the i-th 3D point cloud sample input; N Aug is the augmented number, represents the positive samples of the corresponding category, while all other categories As negative samples, is the number of samples for this task, L InfoNCE is the information-noise contrast estimation loss.

[0078] The classification loss L for task t cls,t It is calculated as follows:

[0079]

[0080] Among them, L ce is the cross entropy loss, p i,t represents the predicted probability of the i-th sample in task t, y i,t is the label of the i-th sample.

[0081] The overall training loss L for task t t It is calculated as follows:

[0082] L t =L cls,t +αL cont,t (4)

[0083] Here, α represents the weight of contrastive learning loss, and in this embodiment, α=1.

[0084] Assume D = {D1, D2, ..., D t ,...,D T} represents a series of 3D object classification tasks, where the tth task D t Contains n t training samples, x i,t is the input sample, y i,t belongs to the label set Y t The corresponding label of . Assume that for t≠t′, Y t With Y t″ There is no intersection between them. After completing the training of the t-th task, the entire neural network model must classify the test sets of all previous tasks {D1, D2, ..., D t D1 is a benchmark task with a large-scale training dataset, while the incremental task D t The samples are small, that is, for t>1, n t <<n1.

[0085] In this embodiment, as the entire neural network model continuously learns multiple 3D object classification tasks, it uses a parameter-level mask adapter to freeze the most relevant parameters of the learned task. This preserves knowledge from previous tasks and mitigates catastrophic forgetting. This also maximizes training possibilities for subsequent tasks, allowing relevant portions of previous knowledge to be leveraged to aid learning of new knowledge, thus balancing the stability and plasticity of the network.

[0086] In this embodiment, it is assumed that the entire neural network is parameterized by weight θ and represented as f(x;θ). The entire neural network is a model determined by the parameter θ. Through training, θ will be adjusted to optimize the performance of the model. The output of the model depends not only on the input x, but also on the parameter θ.

[0087] To protect the prior knowledge of the visual language pre-training model from being destroyed, the parameters of the visual language pre-training model used are kept frozen; each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as a weight score s, where a higher score represents more importance to the current task.

[0088] In this embodiment, the linear layer is a network layer involved in training in the neural network. The input is linearly transformed through a set of trainable weight matrices and bias terms, and a new vector is output. The input of the indicator function is the score of the scoring layer, and the output is a binary mask of 0 or 1. The linear layer score is input into the indicator function, and the output obtained at the top c% positions with the highest score is 1, which means that the parameters with a mask of 1 are more important to the current task, and the rest are 0. The mask is not only used to indicate the optimal parameter subset of the current task in the current task, but also used to set the gradient of the parameters with a mask of 1 to zero during subsequent task training to protect the important parameters of the old task from being changed.

[0089] When training each task, all parameters of the parameter-level mask adapter are first randomly initialized, including the linear layer weights θ and their corresponding weight scores s; the top c% weights with the highest scores represent the parameters most relevant to the current task and are indicated by the function The weight mask assigned to this position is 1, and the rest of the weight masks are set to 0;

[0090] During training, each task generates a mask corresponding to the current task, indicating the most relevant parameters for the current task. The weights whose current task mask is 1 participate normally in training and loss function calculation. The weight scores can be updated synchronously as a trainable parameter layer to optimize the evaluation ability of the weight scores.

[0091] The optimal weighted subset for task t is given by θ t =θ⊙m t Determine, where ⊙ represents the element-wise product, m t is a binary mask.

[0092] During backpropagation, the union of all learned old task masks is obtained through Boolean OR operation As a cumulative binary mask, where ∨ represents the Boolean OR operation, for example, mask m1 = (0, 1, 1, 0) and mask m2 = (1, 0, 1, 0), the OR operation of the two results in the union M2 = (1, 1, 1, 0). The weights in the union with a mask of 1 indicate the most relevant parameters of the previous task. During the training of the current task, the gradients need to be zeroed and not updated to protect these previously learned parameters of the previous task and prevent the subsequent task training from destroying the learned knowledge of the previous task.

[0093] Specifically, when task t is trained, the weight and mask m * t The optimization process is determined by the following formula:

[0094]

[0095] When the subsequent task arrives, the accumulated mask M t-1 To protect the old task weights, the weight score s is updated synchronously to guide the new task-specific mask. The update process is as follows:

[0096]

[0097] where η represents the learning rate of the optimizer. This process allows the network to retain knowledge learned from previous tasks while maintaining flexibility, thus providing intuitive continuous learning capabilities for 3D tasks.

[0098] In order to evaluate the continuous learning ability of the entire neural network model, this embodiment proposes two new evaluation indicators, namely network focus drift and macro forgetting, specifically:

[0099] Network focus drift σ distract It is used to measure the degree of focus drift of the network when continuously learning multiple tasks. It is calculated in a similar way to variance. The higher the value, the greater the drift of network focus when training multiple tasks, and the more difficult it is for the network to maintain stable performance. The specific calculation formula is as follows:

[0100]

[0101] Among them, T is the total number of tasks for network training, MAcc t is the macro accuracy of task t, and its calculation formula is K t is the number of categories in task t, Acc i is the accuracy of category i, NCAcc is the accuracy of novel categories, and its calculation formula is Number of samples predicted correctly for class i, Number of samples for the task.

[0102] Macro-forgetting Δ Macro Macro-forgetting is evaluated for the whole training process, which evaluates the performance change of each incremental task relative to the previous task. The larger the value, the more likely the neural network model is to suffer from catastrophic forgetting when training multiple tasks. The specific formula is as follows:

[0103]

[0104] Using the above two model continuous learning capability evaluation indexes, more comprehensive and reliable evaluation results can be provided for the 3D object classification continuous learning task.

[0105] Reference Figure 2 The design of using parameter-level mask to realize continuous learning is as follows:

[0106] When training the benchmark task, the c% of the weight score indicates the optimal parameter subset of the benchmark task, the task-specific mask is set to 1, and the mask of the remaining weights is set to 0. The weights with mask value 1 can participate in training, and the weight score can be optimized during training to more effectively select the parameters most relevant to the benchmark task.

[0107] When training the incremental task 1, the adapter evaluates the performance of all weights of the linear layer, the c% of the highest score is set to the incremental task 1 specific mask value 1, and is selected as the optimal parameter subset of the task. At this time, the benchmark task specific mask is included in the cumulative mask, and the weight gradient with the cumulative mask value 1 is set to zero to retain the knowledge of the benchmark task.

[0108] When training the incremental task 2, the adapter evaluates the performance of all weights of the linear layer, and selects the c% of the highest score, sets the incremental task 2 specific mask value to 1, and is selected as the optimal parameter subset of the incremental task 2. At this time, the cumulative mask includes the benchmark task specific mask and the incremental task 1 specific mask, and the weight gradient with the cumulative mask value 1 is set to zero to retain the knowledge of the two old tasks. The training of subsequent tasks follows the same process, and the optimal parameters are continuously selected and frozen.

[0109] During training, the network uses the ADAM weight decay optimizer, the batch size is set to 32, the learning rate is set to 1x10 -3 , and the weight decay is set to 1x10 -4 . The temperature coefficient τ of L InfoNCE is set to 0.1 (a lower temperature coefficient makes the model focus more on high similarity samples, and a higher temperature coefficient makes the similarity distribution curve smoother). The training of the benchmark task and each incremental task lasts for 10 and 20 rounds respectively.

[0110] As Figure 3 The learning results of the 3D multi-task continuous learning solution implementation case proposed in the embodiment are shown in the following table: Figure 3 It can be seen that, compared with the method without the parameter-level mask adapter, the 3D continuous learning method based on the parameter-level mask adapter proposed in the embodiment has smaller fluctuations in macro accuracy from the base task (Base Task) to the incremental tasks (Incre-T1, Incre-T2, Incre-T3, Incre-T4, Incre-T5, Incre-T6), and the model is more stable, which has a significant advantage.

[0111] Embodiment Two

[0112] In one or more embodiments, a 3D object classification continuous learning system based on a parameter-level mask adapter is disclosed, comprising:

[0113] A feature extraction module is configured to obtain 3D point cloud data to be tested, and extract 2D features, original 3D features and text features of the 3D point cloud data, respectively.

[0114] A feature fusion module is configured to fuse the 2D features and the 3D features through a parameter-level mask adapter to obtain global features.

[0115] A contrast learning module is configured to perform contrast learning on the global features and the text features to obtain a 3D object classification result.

[0116] In the parameter-level mask adapter, the most relevant parameters of the learned task are frozen during continuous learning of multiple 3D object classification tasks, and the most relevant parameters of the learned task are no longer updated in subsequent tasks.

[0117] The specific implementation modes of the above modules are the same as those in Embodiment One, and will not be described in detail.

[0118] Embodiment Three

[0119] In one or more embodiments, a terminal device is disclosed, comprising a processor and a memory, the processor being configured to implement instructions; the memory is configured to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor to implement the 3D object classification continuous learning method based on the parameter-level mask adapter described in Embodiment One.

[0120] It should be appreciated that in this embodiment, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0121] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0122] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0123] The above describes the specific embodiments of the application in conjunction with the accompanying drawings, but is not a limitation on the protection scope of the application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the application without creative labor are still within the protection scope of the application.

Claims

1. A continuous learning method for 3D object classification based on parameter-level mask adapter, characterized by: include: Obtain the 3D point cloud data to be measured, and extract the 2D features, original 3D features, and text features of the 3D point cloud data respectively; fusing the 2D features and 3D features through a parameter-level mask adapter to obtain a global feature; The global features are compared with the text features to obtain a 3D object classification result; The parameter-level mask adapter freezes the most relevant parameters of the learned tasks when continuously learning multiple 3D object classification tasks, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks; Among them, the parameter-level mask adapter freezes the most relevant parameters of the learned tasks when continuously learning multiple 3D object classification tasks, specifically: Assume that the entire network consists of weights The parameterized representation is , each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as a weight score ; When training each task, we first randomly initialize all parameters of the parameter-level mask adapter, including the linear layer weights and its corresponding weight score The top c% of weights with the highest weight scores represent the parameters most relevant to the current task. Their weight masks are assigned a value of 1, and the remaining weight masks are assigned a value of 0. When training the current task, weights with a mask value of 1 participate in the training. During backpropagation, the masked union of previously learned tasks is obtained through Boolean OR operation, indicating the most relevant weights of all previously learned tasks, and their corresponding gradients are set to zero to ensure that the learned weights are not destroyed during the training of the current task.

2. The continuous learning method for 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that Get the 2D features and text features of 3D point cloud data, specifically: The acquired 3D point cloud data is projected from multiple angles into two sets of depth images at different distances, and then rendered into a set of RGB images. The 2D encoder in the CLIP model is used to extract and aggregate features from the depth and RGB images to obtain 2D features. The text encoder in the CLIP model is used to extract text features.

3. The continuous learning method for 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that The original 3D features of the 3D point cloud data are obtained, specifically: the 3D point cloud data is encoded by a 3D encoder to obtain the original 3D features.

4. The continuous learning method for 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that The parameter-level mask adapter includes a 3D part, a 2D part and a global part; the 3D part is used to process the original 3D features to obtain features aligned with the 2D feature dimension. ; The 2D part is used to combine 2D features with features Perform feature fusion to obtain image features ; The global part is used to and features Perform addition and averaging to obtain the fused global features .

5. The continuous learning method for 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that The global features are compared with the text features to obtain the 3D object classification results, which are specifically: The global features and text features are processed by principal component projection method to obtain the global features that remove redundant information and text features ; Global features to remove redundant information and text features Perform cosine similarity calculation, normalize the calculation results to obtain the predicted probability of each category, and select the category with the highest probability as the 3D object classification result.

6. The continuous learning method for 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that The overall training loss for the 3D object classification task is: ； ； ； in, For the task The classification loss, For the task Contrastive learning loss, represents the weight of contrastive learning loss; is the number of samples for this task, is the augmented number, is the information noise contrast estimation loss, the enhancement function Generate camera direction, It is Samples from Calculated The augmented global features, For the input 3D point cloud samples, represents the positive samples of the corresponding category, while all other categories As negative samples, For the Sample labels, is the cross entropy loss, Indicates a task Middle The predicted probability of a sample.

7. The continuous learning method for 3D object classification based on parameter-level mask adapter according to claim 1, characterized in that Also includes: The continuous learning ability of the entire neural network model is evaluated through two evaluation indicators: network focus drift and macro forgetting; The network focus drift degree is specifically: ； The macro forgetting degree is specifically: ； in, is the total number of tasks for network training, For the task The macro accuracy is calculated as follows: , is the number of categories in task t, For category The accuracy of is the novel class accuracy, and the calculation formula is: , For category The number of samples predicted correctly; is the number of samples for this task.

8. A continuous learning system for 3D object classification based on parameter-level mask adapter, characterized in that include: The feature extraction module is used to obtain the 3D point cloud data to be tested and extract the 2D features, original 3D features and text features of the 3D point cloud data respectively; A feature fusion module, configured to fuse the 2D features and 3D features to obtain a global feature through a parameter-level mask adapter; A contrastive learning module is used to compare and learn the global features with the text features to obtain a 3D object classification result; The parameter-level mask adapter freezes the most relevant parameters of the learned tasks when continuously learning multiple 3D object classification tasks, and the most relevant parameters of the learned tasks are no longer updated in subsequent tasks; Among them, the parameter-level mask adapter freezes the most relevant parameters of the learned tasks when continuously learning multiple 3D object classification tasks, specifically: Assume that the entire network consists of weights The parameterized representation is , each linear layer of the parameter-level mask adapter is assigned a trainable parameter layer of the same dimension to act as a weight score ; When training each task, we first randomly initialize all parameters of the parameter-level mask adapter, including the linear layer weights and its corresponding weight score The top c% of weights with the highest weight scores represent the parameters most relevant to the current task. Their weight masks are assigned a value of 1, and the remaining weight masks are assigned a value of 0. When training the current task, weights with a mask value of 1 participate in the training. During backpropagation, the masked union of previously learned tasks is obtained through Boolean OR operation, indicating the most relevant weights of all previously learned tasks, and their corresponding gradients are set to zero to ensure that the learned weights are not destroyed during the training of the current task.

9. A terminal device comprising a processor and a memory, wherein the processor is used to implement instructions; the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the continuous learning method for 3D object classification based on a parameter-level mask adapter according to any one of claims 1 to 7.

Images