A label-free point cloud classification method based on multimodal contrastive learning

Abstract

This invention discloses an unlabeled point cloud classification method based on multimodal contrastive learning, comprising the following steps: acquiring a target domain point cloud; inputting the target domain point cloud into a trained point cloud classification model to obtain the classification result of the target domain point cloud; wherein, the training steps of the point cloud classification model are as follows: acquiring the target domain point cloud and the source domain point cloud; performing feature processing on the source domain point cloud and the target domain point cloud respectively to obtain a source domain enhanced feature set and a target domain enhanced feature set; wherein, the feature processing includes data augmentation, image modality generation, feature extraction, and feature enhancement; training the pre-constructed point cloud classification model by minimizing the total loss function based on the target domain enhanced feature set and the source domain enhanced feature set to obtain a trained point cloud classification model. By introducing image modalities to learn modal complementarity knowledge, and simultaneously improving the performance of contrastive learning through data augmentation and feature enhancement, the classification accuracy of the point cloud classification method is improved.

Description

Technical Field

[0001] The invention relates to an unlabeled point cloud classification method based on multimodal contrastive learning, and belongs to the technical field of point cloud classification. Background Art

[0002] With the development of 3D scanning technology and sensors, point cloud data has been widely used in fields such as autonomous driving, robot navigation, and 3D reconstruction. However, the acquisition of point cloud data is often affected by the type of sensor, environmental conditions, and data collection methods, resulting in significant inter-domain differences between different datasets. This difference can affect the generalization ability of deep learning models, causing models trained on one dataset to perform poorly on another dataset. Therefore, how to achieve effective model migration between different data distributions has become an important topic of current research.

[0003] When we have trained a point cloud classification model in an environment in one area, we can accurately identify objects such as buildings, roads, and vehicles in that area. However, if we directly apply this trained model to another area, due to the different environmental styles of the two areas, that is, the different feature distributions, the performance of the model will usually drop significantly, and it will not be able to accurately identify objects such as buildings, roads, and vehicles in the other area. This is called domain shift. In order to alleviate this problem, domain adaptation has received more and more attention.

[0004] Domain adaptation is a method of transferring knowledge between different but related source and target domains, aiming to reduce the differences between domains and thus improve the performance of the model in the target domain. In the domain adaptation of point cloud data, the distribution of point cloud data collected under different sensors and environmental conditions in the feature space is significantly different, resulting in poor results when the model trained in the source domain is directly applied to the target domain. Moreover, in many practical applications, the data in the target domain is usually lacking in annotations, which makes it difficult to directly apply supervised learning-based methods.

[0005] At present, the invention patent with publication number CN117454223A discloses a field adaptive method for point cloud classification based on contrastive learning, which only performs data enhancement in the sample space and has the disadvantages of single modality limitation and insufficient feature expression ability. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a label-free point cloud classification method based on multimodal contrastive learning, which learns modal complementary knowledge by introducing image modalities and improves the performance of contrastive learning through data enhancement and feature enhancement, thereby improving the classification accuracy of the point cloud classification method.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] The present invention discloses a label-free point cloud classification method based on multimodal contrastive learning, comprising:

[0009] Obtain target domain point cloud;

[0010] Inputting the target domain point cloud into a trained point cloud classification model to obtain a classification result of the target domain point cloud;

[0011] Wherein, the training method of the point cloud classification model includes:

[0012] Obtaining a target domain point cloud and a source domain point cloud; wherein the target domain point cloud has no label of classification results, and the source domain point cloud has a label of classification results;

[0013] Performing feature processing on the source domain point cloud and the target domain point cloud respectively to obtain a source domain enhanced feature set and a target domain enhanced feature set;

[0014] According to the target domain enhanced feature set and the source domain enhanced feature set, the pre-constructed point cloud classification model is trained by minimizing the total loss function to obtain a trained point cloud classification model; wherein the total loss function includes intra-modal contrast loss, cross-modal contrast loss and cross-entropy loss.

[0015] Furthermore, the feature processing includes:

[0016] Taking the source domain point cloud or the target domain point cloud as the original point cloud;

[0017] According to the original point cloud, data enhancement is performed twice to obtain a first enhanced sample and a second enhanced sample;

[0018] According to the original point cloud, an image modality is generated by projection processing to obtain a corresponding image modality;

[0019] Based on a preset point cloud feature extractor, feature extraction is performed on the first enhanced sample and the second enhanced sample respectively to obtain a first point cloud feature and a second point cloud feature;

[0020] Based on a preset visual encoder, feature extraction is performed on the image modality to obtain image features;

[0021] According to the first point cloud features, the second point cloud features and the image features, feature enhancement is performed by Gaussian noise respectively to obtain the first point cloud enhanced features, the second point cloud enhanced features and the image enhanced features, and then to obtain the source domain enhanced feature set or the target domain enhanced feature set.

[0022] Furthermore, the data enhancement includes the following steps:

[0023] According to the original point cloud, a non-uniform density simulation is performed to obtain a non-uniform density point cloud;

[0024] Performing a cropping process according to the non-uniform density point cloud to obtain a cropped point cloud;

[0025] According to the cropped point cloud, a flipping process is performed to obtain a first enhanced sample or a second enhanced sample.

[0026] Furthermore, the non-uniform density simulation comprises the following steps:

[0027] A first anchor point is randomly set outside the original point cloud;

[0028] Calculating the distance from each point in the original point cloud to the first anchor point and performing normalization processing to obtain normalized data;

[0029] The normalized data is used as a discard probability, and points in the original point cloud are randomly discarded based on the discard probability, thereby obtaining a non-uniform density point cloud.

[0030] Furthermore, the clipping process includes the following steps:

[0031] Randomly select a point inside the non-uniform density point cloud as a second anchor point;

[0032] Calculate the distance from each point in the non-uniform density point cloud to the second anchor point;

[0033] A preset number of points closest to the second anchor point in the non-uniform density point cloud are discarded to obtain a cropped point cloud.

[0034] Furthermore, the image modality generation by projection processing comprises the following steps:

[0035] Multiplying the original point cloud by a preset rotation matrix to obtain an original point cloud of a second perspective;

[0036] According to the original point cloud of the second perspective, project it onto the camera plane using the camera model to obtain the pixel coordinates of the corresponding plane;

[0037] According to the pixel coordinates, the corresponding pixel position in the preset image is set to a predetermined color value to obtain a final image modality.

[0038] Furthermore, the feature enhancement by Gaussian noise comprises the following steps:

[0039] Generate first noise data with the same shape as the first point cloud feature, and add the first noise data to the first point cloud feature to obtain a first point cloud enhanced feature;

[0040] Generate second noise data with the same shape as the second point cloud feature, and add the second noise data to the second point cloud feature to obtain a second point cloud enhanced feature;

[0041] Generate image noise data with the same shape as the image feature, and add the image noise data to the image feature to obtain an image enhancement feature;

[0042] The first noise data or the second noise data or the image noise data is generated by collecting from a normal distribution.

[0043] Furthermore, according to the target domain enhanced feature set and the source domain enhanced feature set, the pre-constructed point cloud classification model is trained by minimizing the total loss function, including the following steps:

[0044] For the first point cloud enhancement feature and the second point cloud enhancement feature corresponding to the same original point cloud in the source domain enhancement feature set or the target domain enhancement feature set, average processing is performed to obtain an average point cloud enhancement feature;

[0045] Calculating intra-modality contrast loss and cross-modality contrast loss according to the source domain enhanced feature set, the target domain enhanced feature set and the average point cloud enhanced features;

[0046] Based on a preset classifier, classify and predict the average point cloud enhancement features to obtain corresponding class distribution prediction values;

[0047] Calculating a cross entropy loss based on the source domain enhanced feature set and the class distribution prediction value;

[0048] According to the intra-modality contrast loss, the cross-modality contrast loss and the cross-entropy loss, a total loss function is obtained;

[0049] Taking minimizing the total loss function as the objective function, the pre-built point cloud classification model is trained to obtain a trained point cloud classification model;

[0050] Among them, the total loss function The expression is as follows:

[0051]

[0052] In the formula, represents the intra-modal contrast loss; represents the cross-modal contrast loss; represents the cross entropy loss.

[0053] Furthermore, the expression of the intra-modal contrast loss is as follows:

[0054] ζ ICont =-log exp(sim( z i , z j ) / τ) ∑ k=1 2N 1 [k≠i] exp(sim( z i , z k ) / τ)

[0055] In the formula, represents the intra-modal contrast loss; represents the logarithmic function; represents the natural exponential function; Represents the similarity calculation function; Represents the first point cloud enhanced feature corresponding to the i-th original point cloud; Representation and The second point cloud enhancement features corresponding to the same original point cloud; represents the temperature coefficient; represents the sum of the number of the first point cloud enhancement features and the second point cloud enhancement features; 1 [k≠i] It means that only when The value is 1 if it is not, the value is 0; Represents the first point cloud enhancement feature or the second point cloud enhancement feature corresponding to the k-th original point cloud;

[0056] The expression of the cross-modal contrast loss is as follows:

[0057] ζ CCont =-log exp(sim( z i avg , h i ) / τ) ∑ m =1 N exp(sim( z i avg , h m ) / τ)+ ∑ m =1 N 1 [ m ≠i] exp(sim( z i avg , z m avg ) / τ)

[0058] In the formula, represents the cross-modal contrast loss; represents the logarithmic function; represents the natural exponential function; Represents the similarity calculation function; Represents the average point cloud enhancement feature of the first point cloud enhancement feature and the second point cloud enhancement feature corresponding to the i-th original point cloud; Representation and Image enhancement features corresponding to the same original point cloud; represents the temperature coefficient; Represents the number of original point clouds, that is, the average number of point cloud enhanced features, that is, the number of image enhanced features; 1 [ m ≠i] It means that only when The value is 1 if it is not, the value is 0; represents the image enhancement feature corresponding to the mth original point cloud, Represents the average point cloud enhanced features corresponding to the m-th original point cloud.

[0059] Furthermore, the expression of the cross entropy loss is as follows:

[0060]

[0061] In the formula, represents the cross entropy loss; Represents the number of original point clouds; represents the cross entropy loss function; Represents the true label value of the nth original point cloud; Represents the class distribution prediction value corresponding to the nth original point cloud.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The label-free point cloud classification method based on multimodal contrastive learning of the present invention, on the one hand, learns modal complementary knowledge by introducing image modalities, and on the other hand, in addition to data enhancement in sample space, also performs feature enhancement in feature space, thereby improving the performance of contrastive learning and thus improving the classification accuracy of the point cloud classification method. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a schematic diagram of the label-free point cloud classification method based on multimodal contrastive learning provided in Example 1;

[0065] Figure 2 is a schematic diagram of training the point cloud classification model provided in Example 1;

[0066] Figure 3 is a training framework diagram of the point cloud classification model provided in Example 1;

[0067] Figure 4 This is a schematic diagram of a confusion matrix obtained by using the conventional domain adaptive classification method provided in Example 2;

[0068] Figure 5 is a schematic diagram of a confusion matrix obtained using the method proposed in Example 1 provided in Example 2;

[0069] Figure 6 This is a schematic diagram of feature distribution obtained by using the conventional domain adaptive classification method provided in Example 3;

[0070] Figure 7 This is a schematic diagram of characteristic distribution obtained using the method proposed in Example 1 and provided in Example 3. DETAILED DESCRIPTION

[0071] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0072] Example 1

[0073] This embodiment 1 provides a label-free point cloud classification method based on multimodal contrastive learning, such as Figure 1and Figure 3 As shown, including:

[0074] Obtain target domain point cloud;

[0075] Input the target domain point cloud into the trained point cloud classification model to obtain the classification result of the target domain point cloud;

[0076] Among them, the training method of the point cloud classification model includes:

[0077] Obtain a target domain point cloud and a source domain point cloud; wherein the target domain point cloud has no labels of classification results, and the source domain point cloud has labels of classification results;

[0078] Feature processing is performed on the source domain point cloud and the target domain point cloud respectively to obtain a source domain enhanced feature set and a target domain enhanced feature set;

[0079] According to the target domain enhanced feature set and the source domain enhanced feature set, the pre-built point cloud classification model is trained by minimizing the total loss function to obtain a trained point cloud classification model; wherein the total loss function includes intra-modal contrast loss, cross-modal contrast loss and cross-entropy loss.

[0080] The technical concept of the present invention is: on the one hand, by introducing image modalities to learn modal complementary knowledge, on the other hand, in addition to data enhancement in the sample space, feature enhancement is also performed in the feature space, thereby improving the performance of contrastive learning and thus improving the classification accuracy of the point cloud classification method.

[0081] Specifically, Figure 2 and Figure 3 As shown, the training steps of the point cloud classification model are as follows:

[0082] Step 1: Obtain the target domain point cloud and the source domain point cloud; the target domain point cloud has no labels of classification results, and the source domain point cloud has labels of classification results.

[0083] Step 2: Perform feature processing on the source domain point cloud and the target domain point cloud respectively to obtain a source domain enhanced feature set and a target domain enhanced feature set; wherein the feature processing includes data enhancement, image modality generation, feature extraction and feature enhancement.

[0084] Specifically, feature processing includes the following steps:

[0085] 2.1. Take the source domain point cloud or the target domain point cloud as the original point cloud p. According to the original point cloud p, perform two data enhancements to obtain the first enhanced sample p1 and the second enhanced sample p2. The specific steps are as follows:

[0086] 2.1.1. Perform non-uniform density simulation based on the original point cloud p to obtain a non-uniform density point cloud. The non-uniform density simulation includes the following steps:

[0087] A first anchor point anchor1 is randomly set outside the original point cloud p;

[0088] Calculate the distance from each point in the original point cloud p to the first anchor point anchor1 and perform normalization to obtain normalized data [0, 1];

[0089] The normalized data [0, 1] is used as the discard probability. After randomly discarding points in the original point cloud p based on the discard probability, the farther the point is from the first anchor point anchor1, the greater the discard probability is, and a non-uniform density point cloud p_nud is obtained.

[0090] 2.1.2. Perform cropping on the non-uniform density point cloud to obtain a cropped point cloud. The cropping process includes the following steps:

[0091] Randomly select a point inside the non-uniform density point cloud p_nud as the second anchor point anchor2;

[0092] Calculate the distance from each point in the non-uniform density point cloud p_nud to the second anchor point anchor2;

[0093] A preset number of points closest to the second anchor point anchor2 in the non-uniform density point cloud p_nud are discarded to obtain a cropped point cloud. In this embodiment, d% of the points closest to anchor2 are discarded, and the value of d is determined according to actual conditions and is not limited here.

[0094] 2.1.3. According to the cropped point cloud, flip it along the xy plane to obtain the final enhanced sample.

[0095] Step 2.1 is performed twice on the original point cloud to obtain the first enhanced sample p1 and the second enhanced sample p2.

[0096] 2.2. Based on the original point cloud p, image modality is generated through projection processing to obtain the corresponding image modality img, which specifically includes the following steps:

[0097] Multiply the original point cloud p by the preset rotation matrix R to obtain the original point cloud p_v of the second perspective;

[0098] According to the original point cloud p_v of the second perspective, the coordinate point (x, y, z) is projected onto the camera plane using the camera model to obtain the pixel coordinates (u, v) of the corresponding plane;

[0099] According to the pixel coordinates (u, v), the corresponding pixel position in the preset image is set to a predetermined color value to obtain the final image modality img.

[0100] 2.3. Based on the preset point cloud feature extractor, feature extraction is performed on the first enhanced sample p1 and the second enhanced sample p2 respectively to obtain the first point cloud feature z1 and the second point cloud feature z2.

[0101] In this embodiment, a point cloud feature extractor DGCNN is used for feature extraction.

[0102] 2.4. Based on the preset visual encoder, feature extraction is performed on the image modality to obtain image feature h3.

[0103] In this embodiment, the CLIP visual encoder is used for feature extraction.

[0104] 2.5. According to the first point cloud feature z1, the second point cloud feature z2 and the image feature h3, feature enhancement is performed by Gaussian noise to obtain the first point cloud enhanced feature za1, the second point cloud enhanced feature za2 and the image enhanced feature, and then the source domain enhanced feature set or the target domain enhanced feature set is obtained.

[0105] Feature enhancement through Gaussian noise includes the following steps:

[0106] Generate first noise data n1 with the same shape as the first point cloud feature z1, add the first noise data n1 to the first point cloud feature z1, and obtain a first point cloud enhanced feature za1;

[0107] Generate second noise data n2 with the same shape as the second point cloud feature z2, add the second noise data n2 to the second point cloud feature z2, and obtain a second point cloud enhanced feature za2;

[0108] Generate image noise data n3 with the same shape as the image feature h3, add the image noise data n3 to the image feature h3, and obtain the image enhancement feature ha3;

[0109] The first noise data n1 or the second noise data n2 or the image noise data n3 is generated by collecting from a normal distribution, and the mean and standard deviation are 0 and 0.1 respectively.

[0110] Step 3: According to the target domain enhanced feature set and the source domain enhanced feature set, the pre-built point cloud classification model is trained by minimizing the total loss function to obtain a trained point cloud classification model;

[0111] The specific steps include:

[0112] 3.1. For the first point cloud enhancement feature za1 and the second point cloud enhancement feature za2 corresponding to the same original point cloud p in the source domain enhancement feature set or the target domain enhancement feature set, average processing is performed to obtain the average point cloud enhancement feature ;

[0113] 3.2. Enhanced feature set based on source domain, target domain and average point cloud , calculate the intra-modality contrastive loss and the cross-modality contrastive loss.

[0114] The expression of intra-modal contrast loss is as follows:

[0115] ζ ICont =-log exp(sim( z i , z j ) / τ) ∑ k=1 2N 1 [k≠i] exp(sim( z i , z k ) / τ)

[0116] In the formula, represents the intra-modal contrast loss; represents the logarithmic function; represents the natural exponential function; Represents the similarity calculation function; Represents the first point cloud enhanced feature za1 corresponding to the i-th original point cloud; Representation and The second point cloud enhanced feature za2 corresponding to the same original point cloud; represents the temperature coefficient; represents the sum of the number of the first point cloud enhanced features za1 and the second point cloud enhanced features za2; 1 [k≠i] It means that only when The value is 1 if it is not, the value is 0; It represents the first point cloud enhancement feature or the second point cloud enhancement feature corresponding to the kth original point cloud. The numerator is the similarity of the positive sample pair, and the denominator is the sum of the similarities of the negative sample pair. Only when the similarity of the positive sample pair is higher and the similarity of the negative sample pair is lower, the intra-modal contrast loss function value will be lower.

[0117] The expression of cross-modal contrast loss is as follows:

[0118] ζ CCont =-log exp(sim( z i avg , h i ) / τ) ∑ m =1 N exp(sim( z i avg , h m ) / τ)+ ∑ m =1 N 1 [ m ≠i] exp(sim( z i avg , z m avg ) / τ)

[0119] In the formula, represents the cross-modal contrast loss; represents the logarithmic function; represents the natural exponential function; Represents the similarity calculation function; Represents the average point cloud enhancement feature of the first point cloud enhancement feature za1 and the second point cloud enhancement feature za2 corresponding to the i-th original point cloud p; Representation and The image enhancement feature ha3 corresponding to the same original point cloud p; represents the temperature coefficient; Represents the number of original point clouds, that is, the average number of point cloud enhancement features, that is, the number of image enhancement features ha3; 1 [ m ≠i] It means that only when The value is 1 if it is not, the value is 0; represents the image enhancement feature ha3 corresponding to the mth original point cloud p, Represents the average point cloud enhancement feature corresponding to the mth original point cloud p. The numerator is the similarity of the positive sample pairs, and the denominator is the sum of the similarities of the negative sample pairs. Only when the similarity of the positive sample pairs is higher and the similarity of the negative sample pairs is lower, the cross-modal contrast loss function value will be lower.

[0120] 3.3. Enhance the features of the average point cloud based on the preset classifier Perform classification prediction and obtain the corresponding class distribution prediction value.

[0121] 3.4. Calculate the cross entropy loss based on the source domain enhanced feature set and the class distribution prediction value. The expression of cross entropy loss is as follows:

[0122]

[0123] In the formula, represents the cross entropy loss; Represents the number of original point clouds p; represents the cross entropy loss function; Represents the true label value of the nth original point cloud p; Represents the class distribution prediction value corresponding to the nth original point cloud p.

[0124] 3.5. According to the intra-modal contrast loss, cross-modal contrast loss and cross-entropy loss, the total loss function is obtained; the total loss function The expression is as follows:

[0125]

[0126] In the formula, represents the intra-modal contrast loss; represents the cross-modal contrast loss; represents the cross entropy loss.

[0127] 3.6. Taking minimizing the total loss function as the objective function, the pre-built point cloud classification model is trained to obtain a trained point cloud classification model.

[0128] The existing technology mainly relies on a single point cloud modality for classification, and fails to make full use of the rich information in the image modality. The present invention introduces the image modality, combines it with the point cloud modality, and learns the complementary knowledge between the modalities, thereby improving the classification performance. For example, the image modality can provide more texture and shape information, which complements the geometric information in the point cloud data, so that the model can understand the characteristics of the object more comprehensively. In addition, the present invention not only performs data enhancement in the sample space, but also in the feature space. By transforming the features, the model can better capture the subtle differences in the data and improve the ability to distinguish different categories. Compared with the existing technology that relies solely on data enhancement, the present invention further introduces feature enhancement, so that the model can learn richer and more recognizable feature representations. Therefore, the present invention significantly improves the performance and adaptability of the point cloud classification model through modal complementarity and feature enhancement.

[0129] Example 2

[0130] In this embodiment 2, the ShapeNet dataset is used as the source domain and the ScanNet dataset is used as the target domain as the experimental setting, and a classification test is performed on the target domain.

[0131] like Figure 4 and Figure 5 As shown, this embodiment uses a confusion matrix for visualization, and the darker the color on the diagonal line, the higher the accuracy. Figure 4 A schematic diagram showing the confusion matrix obtained using the current conventional domain adaptive classification method; Figure 5 A schematic diagram of the confusion matrix obtained using the method proposed in Example 1 is shown. By comparison, it can be seen that Figure 5 The color of the lower right corner of is darker, which shows that the method of the present invention can better reduce the domain gap and achieve better domain adaptation.

[0132] Example 3

[0133] In Example 3, the ScanNet dataset is used as the source domain and the ShapeNet dataset is used as the target domain as the experimental setting, and t-SNE is used to visualize the feature distribution of the target domain. Different colors represent different categories.

[0134] Figure 6 A schematic diagram showing the feature distribution obtained using the current conventional domain adaptive classification method; Figure 7 It is a schematic diagram showing the feature distribution obtained by using the method proposed in Example 1. By comparison, it can be seen that by using the method proposed in the present invention, the feature distribution in the target domain begins to converge, thereby generating clear clusters and effectively reducing the mixing of features of different categories.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A label-free point cloud classification method based on multimodal contrastive learning, characterized in that: include: Obtain target domain point cloud; Inputting the target domain point cloud into a trained point cloud classification model to obtain a classification result of the target domain point cloud; Wherein, the training method of the point cloud classification model includes: Obtaining a target domain point cloud and a source domain point cloud; wherein the target domain point cloud has no label of classification results, and the source domain point cloud has a label of classification results; Performing feature processing on the source domain point cloud and the target domain point cloud respectively to obtain a source domain enhanced feature set and a target domain enhanced feature set; According to the target domain enhanced feature set and the source domain enhanced feature set, the pre-constructed point cloud classification model is trained by minimizing the total loss function to obtain a trained point cloud classification model; wherein the total loss function includes intra-modality contrast loss, cross-modality contrast loss and cross-entropy loss; The feature processing includes: Taking the source domain point cloud or the target domain point cloud as the original point cloud; According to the original point cloud, data enhancement is performed twice to obtain a first enhanced sample and a second enhanced sample; According to the original point cloud, an image modality is generated by projection processing to obtain a corresponding image modality; Based on a preset point cloud feature extractor, feature extraction is performed on the first enhanced sample and the second enhanced sample respectively to obtain a first point cloud feature and a second point cloud feature; Based on a preset visual encoder, feature extraction is performed on the image modality to obtain image features; According to the first point cloud features, the second point cloud features and the image features, feature enhancement is performed by Gaussian noise respectively to obtain the first point cloud enhanced features, the second point cloud enhanced features and the image enhanced features, and then to obtain the source domain enhanced feature set or the target domain enhanced feature set.

2. The label-free point cloud classification method based on multimodal contrastive learning according to claim 1, characterized in that: The data enhancement includes the following steps: According to the original point cloud, a non-uniform density simulation is performed to obtain a non-uniform density point cloud; Performing a cropping process according to the non-uniform density point cloud to obtain a cropped point cloud; According to the cropped point cloud, a flipping process is performed to obtain a first enhanced sample or a second enhanced sample.

3. The label-free point cloud classification method based on multimodal contrastive learning according to claim 2 is characterized in that: The non-uniform density simulation comprises the following steps: A first anchor point is randomly set outside the original point cloud; Calculating the distance from each point in the original point cloud to the first anchor point and performing normalization processing to obtain normalized data; The normalized data is used as a discard probability, and points in the original point cloud are randomly discarded based on the discard probability, thereby obtaining a non-uniform density point cloud.

4. The label-free point cloud classification method based on multimodal contrastive learning according to claim 2, characterized in that: The cutting process comprises the following steps: Randomly select a point inside the non-uniform density point cloud as a second anchor point; Calculate the distance from each point in the non-uniform density point cloud to the second anchor point; A preset number of points closest to the second anchor point in the non-uniform density point cloud are discarded to obtain a cropped point cloud.

5. The label-free point cloud classification method based on multimodal contrastive learning according to claim 1, characterized in that: The image modality generation by projection processing comprises the following steps: Multiplying the original point cloud by a preset rotation matrix to obtain an original point cloud of a second perspective; According to the original point cloud of the second perspective, project it onto the camera plane using the camera model to obtain the pixel coordinates of the corresponding plane; According to the pixel coordinates, the corresponding pixel position in the preset image is set to a predetermined color value to obtain a final image modality.

6. The label-free point cloud classification method based on multimodal contrastive learning according to claim 1, characterized in that: The feature enhancement by Gaussian noise comprises the following steps: Generate first noise data with the same shape as the first point cloud feature, and add the first noise data to the first point cloud feature to obtain a first point cloud enhanced feature; Generate second noise data with the same shape as the second point cloud feature, and add the second noise data to the second point cloud feature to obtain a second point cloud enhanced feature; Generate image noise data with the same shape as the image feature, and add the image noise data to the image feature to obtain an image enhancement feature; The first noise data or the second noise data or the image noise data is generated by collecting from a normal distribution.

7. The label-free point cloud classification method based on multimodal contrastive learning according to claim 1, characterized in that: According to the target domain enhanced feature set and the source domain enhanced feature set, the pre-constructed point cloud classification model is trained by minimizing the total loss function, including the following steps: For the first point cloud enhancement feature and the second point cloud enhancement feature corresponding to the same original point cloud in the source domain enhancement feature set or the target domain enhancement feature set, average processing is performed to obtain an average point cloud enhancement feature; Calculating intra-modality contrast loss and cross-modality contrast loss according to the source domain enhanced feature set, the target domain enhanced feature set and the average point cloud enhanced features; Based on a preset classifier, classify and predict the average point cloud enhancement features to obtain corresponding class distribution prediction values; Calculating a cross entropy loss based on the source domain enhanced feature set and the class distribution prediction value; According to the intra-modality contrast loss, the cross-modality contrast loss and the cross-entropy loss, a total loss function is obtained; Taking minimizing the total loss function as the objective function, the pre-built point cloud classification model is trained to obtain a trained point cloud classification model; Among them, the total loss function The expression is as follows: ， In the formula, represents the intra-modal contrast loss; represents the cross-modal contrast loss; represents the cross entropy loss.

8. The label-free point cloud classification method based on multimodal contrastive learning according to claim 1, characterized in that: The expression of the intra-modal contrast loss is as follows: ， In the formula, represents the intra-modal contrast loss; represents the logarithmic function; represents the natural exponential function; Represents the similarity calculation function; Represents the first point cloud enhanced feature corresponding to the i-th original point cloud; Representation and The second point cloud enhancement features corresponding to the same original point cloud; represents the temperature coefficient; represents the sum of the number of the first point cloud enhancement features and the second point cloud enhancement features; It means that only when The value is 1 if it is not, the value is 0; Represents the first point cloud enhancement feature or the second point cloud enhancement feature corresponding to the k-th original point cloud; The expression of the cross-modal contrast loss is as follows: ， In the formula, represents the cross-modal contrast loss; Represents the average point cloud enhancement feature of the first point cloud enhancement feature and the second point cloud enhancement feature corresponding to the i-th original point cloud; Representation and Image enhancement features corresponding to the same original point cloud; represents the temperature coefficient; Represents the number of original point clouds, that is, the average number of point cloud enhanced features, that is, the number of image enhanced features; It means that only when The value is 1 when it is, otherwise the value is 0; represents the image enhancement feature corresponding to the mth original point cloud, Represents the average point cloud enhanced features corresponding to the m-th original point cloud.

9. The label-free point cloud classification method based on multimodal contrastive learning according to claim 1, characterized in that: The expression of the cross entropy loss is as follows: ， In the formula, represents the cross entropy loss; Represents the number of original point clouds; represents the cross entropy loss function; Represents the true label value of the nth original point cloud; Represents the class distribution prediction value corresponding to the nth original point cloud.

Images