Method for few-shot incremental point cloud classification based on multi-modal feature matching

Abstract

The application discloses a few-shot incremental point cloud classification method based on multi-modal feature matching, which uses three ways of cross-modal feature fusion, image modal enhancement and parallel double branch to realize few-shot incremental point cloud classification; the cross-modal feature fusion is to fuse three-dimensional modal features and image modal features of the point cloud by using cross attention, and to enhance the fused features by using self-attention; the image modal enhancement is to learn an adaptive background color based on the three-dimensional modal features of the point cloud, and to enhance the point cloud foreground in the two-dimensional depth map through background information; the parallel double branch is to distinguish the base class samples and the new class samples by dynamically training a binary classifier, and to process the belonging samples by using two groups of different parameters respectively. The application improves the performance of point cloud classification in the few-shot incremental scene, and provides a low-cost and high-efficiency solution for the industrial fields such as anomaly detection and automatic driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and three-dimensional computer vision, and particularly refers to a few-shot incremental point cloud classification method based on multi-modal feature matching. BACKGROUND

[0002] With the rapid development of three-dimensional sensors and three-dimensional reconstruction and generation technologies, the scale and variety of three-dimensional assets are rapidly growing, and how to efficiently process three-dimensional data has become one of the main bottlenecks in its application. Taking point cloud as an example, as one of the important forms of three-dimensional data, it has the characteristics of disorder, large data volume, and loose structure, so reliably extracting point cloud features is a core challenge for downstream tasks such as classification and segmentation. At the same time, point cloud classification often relies on a large number of labeled data for full-supervised training, which brings great difficulty to emerging fields that lack relevant data sets. In addition, with the rapid increase in the number of point cloud categories, for example, in industrial defect detection, common defect categories such as breakage, cracking, and dirt may not be able to cover all defect categories that appear in subsequent processes, which poses a severe challenge to the generalization ability and continuous learning ability of point cloud classification methods. Therefore, there is an urgent need for a point cloud classification method that can effectively generalize to different fields and has the ability of few-shot incremental learning.

[0003] Traditional point cloud classification methods usually extract point cloud features layer by layer through the construction of a neighborhood graph, and then integrate these features into global features for downstream tasks. However, these methods rely on full-supervised training with labeled data, and each class requires a large number of samples to support, making it difficult to adapt and generalize to few-shot classification tasks. In addition, some works have adopted three-dimensional distillation methods based on multi-modal pre-training, although they have made some progress in zero-shot and few-shot tasks, but their classification stability on new class samples still needs to be improved. In addition, the number of classes and samples of point clouds in actual production applications is often unpredictable, which puts higher requirements on the robustness and continuous learning ability of existing methods. SUMMARY

[0004] The present application aims to overcome the shortcomings and defects of the prior art and proposes a few-shot incremental point cloud classification method based on multi-modal feature matching, which uses a widely used contrastive visual-linguistic pre-training model (CLIP model) to generate point cloud text description features as the prototype features for point cloud classification, and uses the Vision Transformer of CLIP to extract image modal features from the two-dimensional depth map obtained by projecting the point cloud. At the same time, the image modal features and the three-dimensional modal features obtained through the PointNet network are cross-modal fused to enhance the accuracy and generalization of point cloud classification. Finally, a binary classifier is used to distinguish between base class and new class samples, and the feature distance between the two is maximized to achieve effective classification.

[0005] To achieve the above object, the technical scheme provided by the present application is: a few-shot incremental point cloud classification method based on multi-modal feature matching, which realizes few-shot incremental point cloud classification using cross-modal feature fusion, image modal enhancement and parallel double branch, wherein the cross-modal feature fusion uses a cross attention to fuse the three-dimensional modal features and image modal features of the point cloud, and then uses a self-attention to further enhance the fused cross features; the image modal enhancement learns an adaptive background color based on the three-dimensional modal features of the point cloud, and enhances the point cloud foreground in the two-dimensional depth map through additional background information to highlight the contour of the point cloud body; the parallel double branch dynamically trains a binary classifier to maximize the separation of base class samples and new class samples, and uses two different sets of parameters to process the respective samples.

[0006] The specific implementation of the method includes the following steps:

[0007] 1) For each input point cloud, first render the point cloud into a two-dimensional depth map, and use a pre-trained depth map feature extractor to extract the image modal features corresponding to the point cloud from the two-dimensional depth map, then use a PointNet network to process the point cloud data to obtain the three-dimensional modal features of the point cloud;

[0008] 2) Use a cross attention to fuse the three-dimensional modal features and image modal features of the point cloud to obtain cross features, use a self-attention to process the cross features, and introduce a mask ratio to mask the attention weights generated in the self-attention, minimize the cosine similarity difference between the two features generated by the masked and unmasked operations to obtain a reinforced self-attention feature, then generate a background color based on the three-dimensional modal features of the point cloud and overlay it on the two-dimensional depth map for image modal enhancement to obtain a background enhanced image, and use the obtained background enhanced image and the text description features of the point cloud as inputs of a CLIP model to obtain a set of enhanced probability distribution values;

[0009] 3) On each incremental task, dynamically train a binary classifier based on one sample from each class of the base class and few samples from the new class to maximize the boundary between the base class and the new class, and use two different sets of parameters to process the base class and new class samples in parallel double branch;

[0010] 4) Calculate the dot product between the self-attention features of the point cloud and the text description features of the point cloud to obtain a set of probability distributions, fuse the probability distributions with the above enhanced probability distribution values to obtain the final probability distribution, and finally select the maximum value of each point cloud in the above probability distribution as the prediction result.

[0011] Further, the step 1) comprises the following steps:

[0012] 1.1) using a renderer which takes point cloud data as input, rendering a two-dimensional depth map corresponding to the point cloud by projecting the point cloud to a two-dimensional plane:

[0013] I p = Renderer (P)

[0014] wherein P is a three-dimensional point cloud data of shape [32, 1024, 3], wherein 32 is the batch size, representing processing 32 point cloud data at a time, 1024 is the number of points contained in the point cloud, and 3 is the three-dimensional coordinates of each point; Renderer is a renderer for rendering a three-dimensional point cloud into a two-dimensional depth map, I p is a two-dimensional depth map corresponding to the point cloud P;

[0015] 1.2) using a pre-trained depth map feature extractor to extract a depth map feature from the two-dimensional depth map of the point cloud:

[0016] F I = Encoder (I p )

[0017] wherein Encoder is a depth map feature extractor pre-trained based on CLIP Vision Transformer, F I is the obtained point cloud depth map feature, i.e., the image modal feature of the point cloud;

[0018] 1.3) three-dimensional modal feature acquisition: using a PointNet network to process the point cloud data to obtain a three-dimensional modal feature of the point cloud:

[0019] F 3D = PointNet (P)

[0020] wherein F 3D represents the three-dimensional modal feature of the point cloud.

[0021] Further, the step 2) comprises the following steps:

[0022] 2.1) modal feature cross fusion: projecting the three-dimensional modal feature and the image modal feature to the same feature dimension, and using a cross attention to fuse the features of the two modalities together to obtain a cross feature of the point cloud:

[0023] F cross = CrossAttention (F 3D , F I )

[0024] where CrossAttention represents a cross-attention module, which takes the three-dimensional modal feature F 3D as query, and the image modal feature F I as key and value, and computes the cross-attention feature F cross ;

[0025] 2.2) Self-attention feature extraction: a self-attention is used to process the cross-attention feature F cross , while a mask ratio R m is introduced to mask the attention weights generated in the self-attention, and the more representative point cloud features are extracted by comparing the generated self-attention feature and the masked self-attention feature:

[0026]

[0027] where MaskedSelfAttention represents a masked self-attention module, which takes the cross-attention feature F cross and the mask ratio R m as input, R m is a floating-point number with a value range of 0-1, the mask operation represents that when R m is greater than 0, the values of the self-attention weights with a size order of the first R m are set to zero, and the non-mask operation represents that the value of R m is taken as 0, i.e., the attention weights are not modified by the mask, and a pair of masked self-attention features F and self-attention features F self are calculated by the mask and non-mask operations of the masked self-attention, the masked self-attention module learns more representative features by minimizing the cosine similarity difference between F self and F , where Loss s is defined as the cosine similarity difference, cos represents the cosine similarity between two features, with a value of 0-1, is only used to calculate the self-attention loss, and F self is the final self-attention feature;

[0028] 2.3) Generation of background enhancement map: a set of adaptive RGB values are generated based on the three-dimensional modal feature of the point cloud as the background color of the two-dimensional depth map of the point cloud:

[0029] R, G, B = ColorNet(F 3D )

[0030] In the formula, ColorNet is an MLP network that takes the three-dimensional modal features of point cloud as input and generates a set of RGB values ​​between 0 and 1, that is, the values ​​of the three channels R, G and B are all between 0 and 1, corresponding to a background color;

[0031] The two-dimensional depth map of the point cloud I p By blending it with the background color, a background enhancement image is obtained.

[0032]

[0033] 2.4) Enhance the background image and the text description features T corresponding to the point cloud p The input is fed into the CLIP model to obtain the augmentation probability distribution values ​​logits of the point cloud for each category. aux , and have The text description feature T corresponding to the point cloud p Use the following template to generate: "An image of a[class]", where class represents the category to which the point cloud belongs.

[0034] Furthermore, step 3) includes the following steps:

[0035] 3.1) For each incremental task, a binary classifier is first pre-trained based on the sample data from each class of the base class and the small sample data of the incremental task. Then, the binary classifier is used to distinguish whether each sample of the current task belongs to the base class or the new class during the testing phase.

[0036] 3.2) Generate the corresponding position mask for the base class sample, pass the base class data to the base class branch network with frozen parameters for processing, and send the new class data into the current network that is not frozen for processing;

[0037] mask = MG(F 3D )

[0038]

[0039] In the formula, MG represents a binary classifier, mask is a Boolean value that is converted to 0 or 1 when used, where 0 represents that the sample belongs to the new class and 1 represents that the sample belongs to the base class, and X mask The sample is after masking, and then the samples belonging to the base class are sent to the base class network. B Processing is performed, and the new class samples are simultaneously fed into the new class network. N As another branch, the results calculated from the two branches are combined to obtain a set of logits values, which represent the probability distribution of the sample in each category.

[0040] Further, the step 4) comprises the following steps:

[0041] 4.1) Get a set of probability distribution of the point cloud on the current class by calculating the dot product between the self-attention feature F of the point cloud and its corresponding text description feature T self p aux , and then add the result to logits to get a vector with a shape of [32, class'], where 32 is the batch size and class' is the total number of classes of the current task.

[0042] logits = F self · T p + logits aux

[0043] 4.2) Take the maximum value in the class' dimension to get a vector with a shape of

[32] , which represents the classification labels corresponding to the 32 samples, that is, the final classification result.

[0044] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0045] 1. The cross-modal feature fusion includes a cross-attention and a self-attention, which can effectively integrate information from multiple modalities, facilitate further feature matching, and has good reusability and scalability, and can be deployed to other fields involving cross-modal feature fusion.

[0046] 2. The image modality enhancement uses a simple and effective lightweight method to enhance the features of the image modality, which can make more full use of the prior knowledge of CLIP, and also can be deployed as a plug-and-play universal module to other tasks.

[0047] 3. The parallel double branch uses the idea of parallel processing to process data from the base class and the new class using two sets of parameters, thereby effectively improving the performance of the method.

[0048] 4. The present application outperforms the existing best method in multiple few-shot classification tasks, and can be deployed to industrial scenes such as defect detection and automatic driving at low cost, realizing the deep combination of machine learning and industrial production practice. BRIEF DESCRIPTION OF DRAWINGS

[0049] Fig. 1 is a data processing flowchart of the method of the present application.

[0050] Fig. 2 is a principle diagram of cross-modal feature fusion.

[0051] Fig. 3 ​​A schematic diagram for image modality enhancement.

[0052] Fig. 4 A schematic diagram for parallel double branches. DETAILED DESCRIPTION

[0053] The application will be described in further detail below with reference to embodiments and drawings, but the embodiments of the application are not limited thereto.

[0054] As shown in the drawings, Figs. 1 to 4 The embodiment discloses a few-shot incremental point cloud classification method based on multi-modal feature matching, is developed using Python programming language with version number 3.7 on Ubuntu distribution of Linux system, and can run in an environment loaded with a Pytorch deep learning framework. The embodiment can run on a single RTX 4090D graphics card. The method is used to realize few-shot incremental point cloud classification in three ways, namely cross-modal feature fusion, image modality enhancement, and parallel double branches. The cross-modal feature fusion is used to fuse three-dimensional modal features and image modal features of the point cloud through a cross attention, and then use a self-attention to further enhance the fused cross features. The image modality enhancement is used to learn an adaptive background color based on the three-dimensional feature of the point cloud, and to enhance the point cloud foreground in the two-dimensional depth map through additional background information to highlight the outline of the point cloud body. The parallel double branches are used to dynamically train a binary classifier to maximize the separation of base class samples and new class samples, and use two different sets of parameters to process the corresponding samples. The specific implementation of the method includes the following steps:

[0055] 1) For each input point cloud, first render the point cloud into a two-dimensional depth map, and use a pre-trained depth map feature extractor to extract the image modal features corresponding to the point cloud from the two-dimensional depth map, and then use a PointNet network to process the point cloud data to obtain the three-dimensional modal features of the point cloud; including the following steps:

[0056] 1.1) Use a renderer that takes point cloud data as input, and renders a two-dimensional depth map corresponding to the point cloud by projecting the point cloud onto a two-dimensional plane:

[0057] I p = Renderer(P)

[0058] In the formula, P is three-dimensional point cloud data with a shape of [32, 1024, 3], where 32 is the batch size, representing processing 32 point cloud data at a time, 1024 is the number of points contained in the point cloud, and 3 is the three-dimensional coordinates of each point; Renderer is a renderer for rendering three-dimensional point cloud into two-dimensional depth map, and I p is the two-dimensional depth map corresponding to the point cloud P;

[0059] 1.2) Extract image modality features from the two-dimensional depth map of the point cloud using a pre-trained depth map feature extractor:

[0060] F I = Encoder(I p )

[0061] where Encoder is a depth map feature extractor pre-trained based on CLIP Vision Transformer, the version of CLIP used in this embodiment is ViT-B / 32, F I is the obtained point cloud depth map feature, i.e., the image modality feature of the point cloud;

[0062] 1.3) Three-dimensional modality feature acquisition: a PointNet network is used to process the point cloud data to obtain the three-dimensional modality feature of the point cloud:

[0063] F 3D = PointNet(P)

[0064] where F 3D represents the three-dimensional modality feature of the point cloud. The PointNet used in this embodiment includes four one-dimensional convolutional layers, and each two convolutional layers form a convolutional block, and each convolutional block uses batch normalization and ReLU activation function to process the data.

[0065] 2) Cross-modal feature fusion of the three-dimensional modality feature of the point cloud and the image modality feature using a cross-attention to obtain the cross-feature of the point cloud, using a self-attention to process the cross-feature, and introducing a mask ratio to mask the attention weight generated in the self-attention, by minimizing the cosine similarity difference between the two features generated by the mask and non-mask operation, to obtain a reinforced self-attention feature, then generate a background color according to the three-dimensional modality feature of the point cloud, and cover it on the two-dimensional depth map for image modality enhancement to obtain a background enhanced map, and use the obtained background enhanced map and the text description feature of the point cloud as the input of the CLIP model to obtain a set of enhanced probability distribution values; including the following steps:

[0066] 2.1) Cross-modal feature fusion: project the three-dimensional modality feature and the image modality feature to the same feature dimension, and use a cross-attention to fuse the features of the two modalities together to obtain the cross-feature of the point cloud:

[0067] F cross = CrossAttention(F 3D ,F I )

[0068] where CrossAttention represents a cross-attention module, which takes the three-dimensional modal feature F 3D as the query, and the image modal feature F I as the key and the value, and then calculates the cross-attention weight and the cross feature F cross In this embodiment, the cross-attention module takes the three-dimensional feature F 3D as the query (Query), and the depth map feature F I as the key (Key) and the value (Value), and then calculates the cross-attention weight and the cross feature F cross The cross-attention is regularized by dropout with a value of 0.1.

[0069] 2.2) Self-attention feature extraction: a self-attention is used to process the cross feature F cross , and a mask ratio R m is introduced to mask the attention weight generated in the self-attention, and the more representative point cloud feature is extracted by comparing the generated self-attention feature and the masked self-attention feature:

[0070]

[0071]

[0072] where MaskedSelfAttention represents a masked self-attention module, which takes the cross feature F cross and the mask ratio R m as the input, R m is a floating-point number with a value range of 0-1, the mask operation represents that when R m is greater than 0, the values of the self-attention weight size sorted in the front R m are set to zero, and the non-mask operation represents that the value of R m is taken as 0, that is, the mask is not used to modify the attention weight; a pair of masked self-attention features F and self-attention features F self are calculated through the mask and non-mask operations of the masked self-attention, the masked self-attention module learns more representative features by minimizing the cosine similarity difference between F self and F , where Loss s is defined as the cosine similarity difference, cos represents the cosine similarity between two features, and the value is 0-1, is only used to calculate the self-attention loss, and F self is the final self-attention feature;

[0073] 2.3) Background enhanced map generation: based on the three-dimensional modal features of the point cloud, a set of adaptive RGB numerical values are generated as the background color of the two-dimensional depth map of the point cloud:

[0074] R,G,B = ColorNet(F 3D )

[0075] In the formula, ColorNet is an MLP network, which takes the three-dimensional modal features of the point cloud as input, generates a set of RGB values with a value range of 0-1, i.e. the values of R, G and B three channels are 0-1, corresponding to a background color;

[0076] The two-dimensional depth map I p of the point cloud is fused with the background color to obtain the background enhanced map

[0077]

[0078] 2.4) Input the background enhanced map and the text description feature T p corresponding to the point cloud into the CLIP model to obtain the enhanced probability distribution value logits of the point cloud on each category aux , and wherein the text description feature T p corresponding to the point cloud is generated using the following template: “An image of a [class]”, wherein class represents the category to which the point cloud belongs, for example, for a point cloud belonging to the car category, the corresponding text description is: “An image of a car”. The CLIP version used in this example is ViT-B / 32, which contains the Vision Transformer and Text Encoder parameters consistent with this version.

[0079] 3) On each incremental task, a binary classifier is dynamically trained according to one sample from each class of the base class and a few samples from the new class to maximize the boundary between the base class and the new class, and two different sets of parameters are used to process the base class and new class samples in parallel double branch; including the following steps:

[0080] 3.1) On each incremental task, first, a binary classifier MG is pre-trained according to one sample from each class of the base class and a few samples of the incremental task, and then the binary classifier is used to distinguish in the test stage whether each sample of the current task belongs to the base class or the new class. In this example, the binary classifier takes the three-dimensional modal features of the point cloud as input, and is optimized by calculating the binary cross-entropy loss between the generated mask and the true value;

[0081] 3.2) When the MG training is completed, it generates a corresponding mask for the input sample, passes the base class data to the base class branch network with frozen parameters for processing, and sends the new class data to the current network that is not frozen for processing;

[0082] mask=MG(F 3D )

[0083]

[0084] wherein MG represents a binary classifier, mask is a Boolean value which is converted to 0 or 1 in use, wherein 0 represents that the sample belongs to the new class, and 1 represents that the sample belongs to the base class, X mask is the sample after the mask operation, then the sample belonging to the base class is sent to the base class network Network B for processing, and the new class sample is sent to the new class network Network N for processing. As another branch parallel processing, finally, the results calculated by the two branches are summarized to obtain a set of logits values representing the probability distribution of the sample in each class. In this embodiment, the base class network Network B freezes its parameters after training, while the new class network Network N keeps all its parameter training configurations unchanged.

[0085] 4) Calculate the dot product between the self-attention features of the point cloud and the text description features of the point cloud to obtain a set of probability distributions, fuse the probability distributions with the above enhanced probability distribution values to obtain the final probability distribution, and finally select the maximum value of each point cloud in the above probability distribution as the prediction result. Including the following steps:

[0086] 4.1) Calculate the dot product between the self-attention features F self of the point cloud and the corresponding text description features T p to obtain a set of probability distributions of the point cloud in the current class, then add the result to logits aux to obtain a vector with a shape of [32, class'], wherein 32 is the batch size, and class' is the total number of classes of the current task;

[0087] logits=F self ·T p +logits aux

[0088] 4.2) Take the maximum value in the class' dimension to obtain a vector with a shape of

[32] , representing the classification labels corresponding to the 32 samples, that is, the final classification result.

[0089] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A few-sample incremental point cloud classification method based on multimodal feature matching, characterized in that, This method employs three approaches to achieve incremental point cloud classification with few samples: cross-modal feature fusion, image modality enhancement, and parallel two-branch. Cross-modal feature fusion uses a cross-attention mechanism to fuse the 3D modal features of the point cloud and the image modality features, followed by self-attention to further enhance the fused cross-features. Image modality enhancement learns an adaptive background color based on the 3D modal features of the point cloud, enhancing the foreground of the point cloud in the 2D depth map with additional background information to highlight the point cloud's outline. Parallel two-branch involves dynamically training a binary classifier to maximize the distinction between base class samples and new class samples, and using two different sets of parameters to process the respective samples. The specific implementation of this method includes the following steps: 1) For each input point cloud, the point cloud is first rendered into a two-dimensional depth map, and the image modal features corresponding to the point cloud are extracted from the two-dimensional depth map using a pre-trained depth map feature extractor. Then, a PointNet network is used to process the point cloud data to obtain the three-dimensional modal features of the point cloud. 2) Use a cross-attention method to fuse the 3D modal features of the point cloud with the image modal features to obtain the cross features of the point cloud. Use a self-attention method to process the cross features. At the same time, introduce a mask ratio to mask the attention weights generated in the self-attention method. By minimizing the cosine similarity difference between the two features generated by the masked and unmasked operations, an enhanced self-attention feature is obtained. Then, generate a background color based on the 3D modal features of the point cloud and overlay it on the 2D depth map to perform image modality enhancement to obtain a background enhancement map. Use the obtained background enhancement map and the text description features of the point cloud as input to the CLIP model to obtain a set of enhancement probability distribution values. 3) On each incremental task, a binary classifier is dynamically trained based on only one sample from each of the base classes and a small number of samples from the new classes to maximize the boundary between the base classes and the new classes. Two different sets of parameters are used to process the base class and new class samples in parallel dual-branch processing. 4) Calculate the dot product between the self-attention features of the point cloud and the text description features of the point cloud to obtain a set of probability distributions. Fuse this probability distribution with the above-mentioned enhanced probability distribution values ​​to obtain the final probability distribution. Finally, select the maximum value of each point cloud in the above probability distribution as the prediction result.

2. The method for few-sample incremental point cloud classification based on multimodal feature matching according to claim 1, characterized in that, Step 1) Includes the following steps: 1.1) Use a renderer that takes point cloud data as input and renders a 2D depth map of the point cloud by projecting the point cloud onto a 2D plane: IN p =Renderer(P) In the formula, P represents the 3D point cloud data of shape [32, 1024, 3], where 32 is the batch size, representing the processing of 32 point cloud data at a time, 1024 is the number of points contained in the point cloud, and 3 is the 3D coordinate of each point; Renderer is the renderer used to render the 3D point cloud into a 2D depth map, and I... p This is the two-dimensional depth map corresponding to point cloud P; 1.2) Extract depth map features from the 2D depth map of the point cloud using a pre-trained depth map feature extractor: F I =Encoder(I p ) In the formula, Encoder is a depth map feature extractor pre-trained based on CLIP Vision Transformer, and F... I The resulting point cloud depth map features are also known as the image modal features of the point cloud. 1.3) 3D Modal Feature Acquisition: A PointNet network is used to process the point cloud data to obtain the 3D modal features of the point cloud. F 3D =PointNet(P) In the formula, F 3D The three-dimensional modal features representing point clouds.

3. The method for few-sample incremental point cloud classification based on multimodal feature matching according to claim 2, characterized in that, Step 2) includes the following steps: 2.1) Modal Feature Cross-Fusion: 3D modal features and image modal features are projected onto the same feature dimension, and a cross-attention mechanism is used to fuse the features of the two modalities together, resulting in the cross-features of the point cloud. F cross =CrossAttention(F 3D ,F I ) In the formula, CrossAttention represents the cross-attention module, which uses the three-dimensional modal features F of the point cloud. 3D As a query, the image modal feature F I As keys and values, the cross-feature F is calculated. cross ; 2.2) Self-attention feature extraction: Using a self-attention pair for cross-feature F cross Processing is performed, and a masking ratio R is introduced. m The attention weights generated in the self-attention process are masked, and more representative point cloud features are extracted by comparing the generated self-attention features with the masked self-attention features. In the formula, MaskedSelfAttention represents a masked self-attention module, which uses cross-feature F cross and the mask ratio R m As input, R m Given a floating-point number ranging from 0 to 1, the masking operation represents when R... m When R is greater than 0, sort the self-attention weights by their magnitude. m Setting the value to zero, a non-mask operation represents setting R... m The value is 0, meaning no mask is used to modify the attention weights; a set of paired masked self-attention features are calculated through masking and non-masking operations of masked self-attention. and self-attention feature F self The mask self-attention module minimizes and F self The cosine similarity difference between them is used to learn more representative features, where Loss s Defined as cosine similarity difference, where cos represents the cosine similarity between two features, with a value ranging from 0 to 1. Used only for calculating self-attention loss, F self This is the final self-attention feature; 2.3) Background Enhancement Map Generation: Based on the 3D modal features of the point cloud, a set of adaptive RGB values ​​is generated as the background color of the 2D depth map of the point cloud. R,G,B=ColorNet(F 3D ) In the formula, ColorNet is an MLP network that takes the three-dimensional modal features of point cloud as input and generates a set of RGB values ​​between 0 and 1, that is, the values ​​of the three channels R, G and B are all between 0 and 1, corresponding to a background color; The two-dimensional depth map of the point cloud I p By blending it with the background color, a background enhancement image is obtained. 2.4) Enhance the background image and the text description features T corresponding to the point cloud p The input is fed into the CLIP model to obtain the augmentation probability distribution values ​​logits of the point cloud for each category. aux , and have The text description feature T corresponding to the point cloud p Use the following template to generate: "An image of a[class]", where class represents the category to which the point cloud belongs.

4. The few-sample incremental point cloud classification method based on multimodal feature matching according to claim 3, characterized in that, Step 3) includes the following steps: 3.1) For each incremental task, a binary classifier is first pre-trained based on the sample data from each class of the base class and the small sample data of the incremental task. Then, the binary classifier is used to distinguish whether each sample of the current task belongs to the base class or the new class during the testing phase. 3.2) Generate the corresponding position mask for the base class sample, pass the base class data to the base class branch network with frozen parameters for processing, and send the new class data into the current network that is not frozen for processing; mask=MG(F 3D ) In the formula, MG represents a binary classifier, mask is a Boolean value that is converted to 0 or 1 when used, where 0 represents that the sample belongs to the new class and 1 represents that the sample belongs to the base class, and X mask The sample is after masking, and then the samples belonging to the base class are sent to the base class network. B Processing is performed, and the new class samples are simultaneously fed into the new class network. N As another branch, the results calculated from the two branches are combined to obtain a set of logits values, which represent the probability distribution of the sample in each category.

5. The few-sample incremental point cloud classification method based on multimodal feature matching according to claim 4, characterized in that, Step 4) includes the following steps: 4.1) Calculate the self-attention feature F of the point cloud self and its corresponding text description feature T p The dot product between the two values ​​yields a probability distribution of the point cloud for the current category. The result is then multiplied by logits. aux The sums result in a vector of shape [32, class'], where 32 is the batch size and class' is the total number of categories in the current task. logits=F self ·T p +logits aux 4.2) Take the maximum value in the class' dimension to obtain a vector of shape [32], which represents the classification label corresponding to the 32 samples, that is, the final classification result.

Images