Point cloud self-supervised learning network model based on comparative learning and training method

Abstract

The invention provides a point cloud self-supervised learning network model based on comparative learning and a training method, and can solve the problems that a traditional model with a supervised mode depends on data with labels in the training stage, manufacturing of point cloud data labels needs to consume a large amount of manpower and material resources, mistakenly labeled data interferes with the training process, and the training efficiency is high. And the final representation learning performance of the model is influenced. The network model comprises a learnable point cloud enhancement module used for performing global transformation and local transformation in a learnable manner according to different types of point cloud samples to realize data enhancement; the refined feature prediction module is used for carrying out feature mutual prediction between point cloud sample pairs in combination with an attention mechanism and improving the feature discrimination degree of the network model for different types of point clouds; and the adaptive loss function is used for continuously optimizing the network model through a gradient feedback mechanism. According to the method, characterization learning tasks such as point cloud classification and segmentation can be efficiently completed, and the obtained effect is better than that of the most advanced supervised training method.

Description

Technical Field

[0001] The present invention relates to the field of point cloud image processing and deep learning, and in particular to a point cloud self-supervised learning network model and a training method based on contrastive learning. Background Art

[0002] With the popularization of sensor devices such as LiDAR and RGB-D cameras, the collection and acquisition of 3D data has become simple, which has greatly promoted the widespread application of 3D data. The three-dimensional objects and subjects collected by these 3D sensor devices are represented in the form of point clouds. Academic research on three-dimensional point clouds is mainly divided into traditional machine learning methods and deep learning-based methods. Traditional machine learning methods mainly use methods such as octrees and KD trees to represent and learn point clouds. They are far behind deep learning methods in various accuracy indicators, and they cannot achieve end-to-end intelligent processing. Deep learning technology can well overcome the various shortcomings of traditional machine learning, effectively improve the accuracy indicators of various point cloud tasks, and achieve end-to-end processing, etc., and has therefore become a research hotspot in recent years. Among the deep learning methods, there are three categories: voxel-based, multi-view, and direct point cloud.

[0003] The voxel-based method mainly rasterizes the original point cloud into 3D voxels, and then uses the 3D CNN network to further extract features, and then processes the subsequent tasks of the point cloud. Point-Voxel normalizes the original point cloud, aggregates features based on voxels, and then de-voxels. Its processing process can operate on each point, and the extracted single point features are used for subsequent point cloud representation learning tasks. NormalNet can consider features from a higher level, use the normal vector of the object surface as input, and give the reflection convolution kernel cascade (Reflection-Convolution-Cascade) model for visual task stratification. This method can extract distinguishable features for three-dimensional visual tasks. VB-Net starts with the network model. After the point cloud is voxelized and undergoes operations such as three-dimensional convolution and pooling, BLS (Broad Learning System) is used to achieve refined feature regression, and it is used for high-precision 3D point cloud classification and segmentation. It is worth noting that the original point cloud inevitably requires complicated preprocessing operations in the process of rasterizing into voxels. At the same time, voxels also have cubic spatial complexity, which makes the calculation and storage cost too high when using 3D CNN for processing. Therefore, it is necessary to reduce the time and space overhead by reducing the resolution of voxels, but this in turn reduces the evaluation performance of the model.

[0004] The multi-view based method projects the point cloud from different perspectives into multiple 2D views, and then uses the existing mature 2D CNN network to extract local features and fuse global features for subsequent processing of all-directional point cloud tasks. MVCNN first uses a virtual camera to project 3D objects into multiple 2D views, and then uses 2D CNN to extract local features, and uses global maximum pooling and average pooling to fuse global features for point cloud classification. However, this method only considers a single view and does not explore the possible relationship between views. In order to model the hierarchical association between discriminative 3D shape descriptors, GVCNN uses a group view convolutional neural network to better explore and utilize the inherent hierarchical association and distinguishability between views for classification. PointView-GCN changes the operation method of traditional CNN and proposes a multi-level graph convolutional network to hierarchically aggregate the shape features of point clouds and encode the relationship between views for subsequent high-precision point cloud classification. However, it requires a large number of views to construct the hierarchical relationship of the graph. In fact, the correlation within the view cannot be ignored. Based on this understanding, MHBN takes the image block to image block similarity measurement as the starting point and utilizes the relationship between polynomial kernel and bilinear pooling. It uses bilinear pooling to aggregate local convolution features and coordinate different components within the view to obtain more discriminative features. Considering the good performance of Transformer in the field of 2D computer vision, MVT introduces it into the multi-view point cloud processing task, so that each image block has a global receptive field and realizes information interaction between view blocks. Based on the above multi-view fixed projection method, related research focuses more on the improvement of view selection methods. FSDCNet uses a fixed plus random viewpoint projection method. The introduction of random viewpoints can effectively avoid the overfitting problem that may exist in the views obtained by traditional fixed viewpoints. SimpleView projects the point cloud onto six orthogonal planes to obtain the most representative 2D multi-views within a limited range, thereby creating a sparse depth image and controlling the orthogonal auxiliary factors related to the network architecture to extract effective view features. MVTN finds the best viewpoint for 3D shape recognition through regression, and obtains multiple 2D views based on differential rendering, so that the extracted features are more representative and can better complete the task of point cloud classification. The multi-view based method can achieve better accuracy indicators overall, but it takes a high cost to render the original point cloud into multiple 2D views. At the same time, the method of selecting the viewing angle and the number of views also affect the classification performance of the model. In addition, the most critical thing is that the multi-view method is only widely used in point cloud classification scenarios, while it is difficult to apply to real-time scenarios such as point cloud segmentation and target detection. This also limits the application scope of the multi-view method to a certain extent.

[0005] The mainstream method of point cloud representation learning is based on direct point cloud processing, that is, the obtained point cloud data is directly processed for subsequent point cloud tasks. PointNet lays the foundation for direct point cloud processing, which uses multi-layer perceptrons to extract the features of each point in the point cloud. At the same time, the network model uses the symmetric function of maximum pooling to fuse global features, but it has the disadvantage of serious loss of local detail features. In order to solve this problem, based on PointNet, PointNet++ uses the farthest point sampling (FPS) and the nearest neighbor algorithm (KNN) to obtain several local point cloud areas on the original point cloud. The model uses the PointNet method to extract local features and perform global feature fusion, achieving a good point cloud classification and segmentation effect. However, due to the use of MSG (Multi-scale grouping) and MRG (Multi resolution grouping) for local feature extraction, the inference speed of the model is slowed down. Unlike methods such as PointNet++ that only use multi-layer perceptrons to aggregate the features of local neighborhood points, DGCNN uses an edgeconv neural network module. This module dynamically calculates the Euclidean distance from the neighborhood point to the center point in the form of graph convolution for each layer in the network. It can capture the potential long-distance semantic features in the original embedding, but it has the problem of limited feature information extracted from local areas. CurveNet uses a long-distance feature aggregation method for point cloud shape analysis to solve this problem. It aggregates arcs of a fixed number and length by guided walks on the point cloud isomorphic graph, thereby supplementing the overall features of the point cloud with additional geometric information. It has achieved advanced results in multiple point cloud analysis tasks, but the scalability of the model and the efficiency of convolution still need to be further improved. Methods such as RPNet, PAConv, and Point Voxel Transformer have further improved the performance indicators of point cloud classification and segmentation through extension modules and adaptive convolution kernels. RPNet includes an efficient and scalable module, the group relationship aggregator, which weights feature sets and calculates group features through geometric and semantic relationships. A large number of experiments have demonstrated the effectiveness of the model in various point cloud processing tasks and its robustness to noise. PAConv applies an effectively positioned adaptive convolution, constructs the convolution kernel by dynamically combining the basic weight matrices in the Weight Bank, and learns the correlation coefficient from the position of the point through ScoreNet.Point Voxel Transformer uses two module branches of voxels and points to process the original point cloud, proposes a sparse window attention (SWA) module to obtain coarse-grained local features, and introduces a relative attention module (RA) to extract refined features for point cloud learning tasks. However, the calculation of these algorithm models has become more complicated due to the introduction of extension modules, and the overhead related to storage costs has also increased further. Therefore, in order to reduce the complexity of the algorithm model, PointMLP proposes a multi-layer perceptron network with residuals, and standardizes the point cloud features through a geometric affine transformation module, which can achieve high classification and segmentation indicators without complex local feature extraction. However, as a supervised learning method, PointMLP still needs to rely on specific data sets and their label data. Therefore, in response to the shortcomings of traditional supervised learning methods, IAE (Implicit AutoEncoder) introduces self-supervised learning methods into 3D point cloud processing, by outputting continuous representations of different point cloud samples under the same model, and giving priority to the sample variants discarded by the encoder, thereby introducing more useful features for spatial learning. However, it requires additional preprocessing steps on the raw point cloud to obtain implicit representation training labels.

[0006] In summary, the direct point cloud-based method uses the point cloud as the input of the algorithm model without a series of complicated preprocessing operations. Therefore, in comparison, its efficiency is the highest among the three different types of methods. At the same time, it can also adapt well to different types of point cloud processing tasks and achieve very high performance indicators. However, since most of the current direct point cloud-based methods use supervised learning, they rely on a large amount of labeled point cloud data. This brings about the two main disadvantages: on the one hand, it requires a lot of manpower and material resources to manually label the data; on the other hand, the point cloud data obtained by various means may have low recognition, which affects the correct manual labeling of the data. These incorrectly labeled data will interfere with the training of the model to a certain extent. Summary of the invention

[0007] The present invention is made to solve the above-mentioned problems, and aims to provide a point cloud self-supervised learning network model and training method based on contrastive learning.

[0008] The present invention provides a point cloud self-supervised learning network model based on contrastive learning, which has the following characteristics, including: a learnable point cloud enhancement module, which is used to perform global and local transformations in a learnable manner according to point cloud samples of different categories to achieve data enhancement and optimization; a refined feature prediction module, which is used to perform feature mutual prediction between point cloud sample pairs in combination with an attention mechanism to improve the feature discrimination of the network model for point clouds of different categories; and an adaptive loss function, which is used to continuously optimize the network model through a gradient feedback mechanism to avoid model collapse.

[0009] The present invention also provides a training method for a point cloud self-supervised learning network model based on contrastive learning, which has the following characteristics and includes the following steps: step S1, pre-training the network model; step S2, fine-tuning and testing the trained network model on downstream tasks; wherein, in step S1, the network model is continuously optimized through an adaptive loss function during the pre-training process, and the pre-training includes the following sub-steps: step S1-1, using a learnable point cloud enhancement module to perform data enhancement on the original point cloud from both global and local perspectives in a learnable manner; step S1-2, first using a weight-sharing encoder to extract features from the original point cloud and the transformed point cloud, and then using a refined feature prediction module to perform feature mutual prediction between the two point cloud samples to achieve self-learning of point cloud representation.

[0010] Functions and Effects of the Invention

[0011] According to the point cloud self-supervised learning network model and training method based on contrastive learning involved in the present invention, the network model can directly compare and learn point cloud samples through itself, and directly apply self-supervised contrastive learning to characterization learning tasks such as classification and partial segmentation of point cloud samples without the need for additional preprocessing steps or auxiliary information. Specifically, the network model includes a learnable point cloud enhancement module, a refined feature prediction module, and an adaptive loss function.

[0012] Among them, in point cloud contrast learning, compared with traditional data enhancement, only global or local random transformations are performed, and the particularity of specific point cloud samples is not taken into account, and the enhancement effect is limited. The learnable point cloud enhancement module can perform specific global and local transformations according to different point cloud samples, and can obtain more superior point cloud enhancement effects, which is helpful to train a more generalized and robust network model.

[0013] In order to improve the encoder's feature discrimination of different categories of point clouds, a refined feature prediction module is designed, which can combine the attention mechanism to enable the encoder to learn the most critical feature information from different point cloud sample pairs, thereby improving the accuracy of tasks such as point cloud classification and part segmentation.

[0014] If the point cloud enhancement is too large, the transformed point cloud will completely lose the geometric shape and structure of the original point cloud. If the point cloud enhancement is too small, the similarity of the point cloud sample pairs will be too high. Both will cause the model to collapse during training. Therefore, in order to strike a balance between point cloud enhancement and contrast representation learning and achieve the best learning effect, an adaptive loss function is designed, which optimizes the entire network model through a gradient feedback mechanism to avoid network model collapse.

[0015] This network model and training method were also compared with encoder models such as PointNet, PointNet++, DGCNN, and PointMLP. A large number of experiments have proved that the effect obtained using this training method is better than the existing advanced supervised training methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a general framework diagram of the PointSL model in an embodiment of the present invention, wherein (a) is a schematic diagram of the pre-training of the PointSL model, and (b) is a schematic diagram of fine-tuning and testing the PointSL model on downstream tasks;

[0017] Figure 2 is a schematic diagram of an LPA module in an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of a self-learning process of a single point cloud feature in an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of an FFP module in an embodiment of the present invention;

[0020] Figure 5 is a schematic diagram comparing the classification accuracy obtained by using a traditional supervised model training and a PointSL model training on ModelNet40 in an embodiment of the present invention;

[0021] Figure 6 The following is a comparison diagram of the visualization of the confusion matrix of traditional supervised training and training using the PointSL model. (a) is a visualization of the confusion matrix of traditional supervised training, and (b) is a visualization of the confusion matrix of training using the PointSL model.

[0022] Figure 7 is a segmentation visualization diagram of a simple category point cloud sample in an embodiment of the present invention;

[0023] Figure 8 is a segmentation visualization diagram of a difficult category point cloud sample in an embodiment of the present invention;

[0024] Fig. 9This is a schematic diagram comparing the loss functions of the traditional random enhancement method and the LPA module on four models. DETAILED DESCRIPTION

[0025] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is specifically described in the following embodiments in conjunction with the accompanying drawings.

[0026] Example

[0027] like Figure 1 As shown, this embodiment provides a point cloud self-supervised learning network model based on contrastive learning and a training method thereof.

[0028] The point cloud self-supervised learning network model based on contrastive learning (hereinafter referred to as the "PointSL model") is used to efficiently complete representation learning tasks such as point cloud classification and segmentation. It mainly includes a learnable point cloud enhancement module (hereinafter referred to as the "LPA module"), a refined feature prediction module (hereinafter referred to as the "FFP module"), and an adaptive contrast loss function.

[0029] Among them, the LPA module is designed for data enhancement of the original point cloud. It can perform specific transformations on a single point cloud sample or a local area of ​​a single point cloud sample to achieve data enhancement and optimization, which can greatly improve the enhancement effect of the point cloud. The FFP module is designed to further improve the encoder's ability to distinguish features of different categories. It combines the attention mechanism to perform feature mutual prediction between point cloud sample pairs, thereby continuously improving the discrimination of the PointSL model. The adaptive loss function is used to continuously optimize the PointSL model through a gradient feedback mechanism, which helps to pre-train an encoder model with better generalization and higher accuracy.

[0030] The training method includes the following steps: Step S1, pre-training the PointSL model, see Figure 1 (a); Step S2: fine-tune and test the trained PointSL model on downstream tasks, see Figure 1 (b). In step S1, pre-training includes the following sub-steps: step S1-1, using the LPA module to perform data enhancement on the original point cloud from both global and local perspectives in a learnable manner, and continuously improving the enhancement effect of the point cloud; step S1-2, first use a weight-sharing encoder to extract features from the original point cloud and the transformed point cloud, and then use the FFP module to perform feature mutual prediction between the two point cloud samples to achieve self-learning of point cloud representation. In step S2, for the point cloud classification task, a fully connected layer is set to evaluate the performance index, and for the point cloud segmentation task, a decoder is set to evaluate the performance index.

[0031] The LPA module, FFP module, and adaptive loss function are described in detail below.

[0032] The contrastive learning method has high requirements for data enhancement, and most of the existing point cloud data random enhancement methods lack specific transformations for single point cloud data, which results in limited effect of point cloud enhancement. PointAugment uses a multi-layer perceptron to enhance and optimize specific point cloud data, but it does not consider data enhancement and optimization of local specific areas of a single point cloud. Therefore, the LPA module is designed in this solution to solve this problem. The purpose of step S1-1 is to use the LPA module to perform data enhancement on the original point cloud.

[0033] The function of the LPA module is: for global point cloud data enhancement, the LPA module generates the rotation and scale transformation matrix M for different point clouds P in a learnable way r , jitter matrix M j For data enhancement in a local specific area, the LPA module generates the corresponding mask matrix M m Finally, the LPA module combines the original point cloud P with M r 、M j 、M m Perform specific transformation operations to generate the final point cloud P t .

[0034] Among them, Figure 2 As shown in the schematic diagram of the LPA module, the rotation and scale transformation matrix M r It is used to rotate and scale the original point cloud in a learnable way. The generation steps are as follows:

[0035] Step 1: Obtain high-dimensional features F h . For the original point cloud P, a multi-layer perceptron (MLPs) 1 is used to extract its features and obtain the feature F. Among them, (MLPs) 1 consists of two 64-channel one-dimensional convolutions, one 128-channel one-dimensional convolution, and one 1024-channel one-dimensional convolution. The one-dimensional convolution layers at each level include BN layers and activation functions ReLU, which ensure the convergence stability and nonlinear expression ability of the features. The multi-layer perceptrons used in the following text are also constructed based on the above method. The specific implementation details can be expressed as formulas (1) and (2):

[0036]

[0037]

[0038] in, p i Represents any point in the point cloud P, which includes Three coordinate dimensions, so p i can be regarded as a three-dimensional vector, that is MLP j It is a combination of multiple convolutional layers, BatchNorm layers, and activation function layers. in represents the number of input channels, c out Represents the number of output channels. σ represents the nonlinear activation function layer, and the ReLU function is used here.

[0039] Step 2: Generate point cloud global feature F g . The extracted high-dimensional feature F containing rich detail information h Finally, use the maximum pooling to get the global features It can be approximately regarded as the global representation of the current point cloud P. For details, see formula (3):

[0040] F g =maxpool(F h ) (3)

[0041] Step 3: Feature concatenation operation. Use Gaussian distribution to generate random noise features To ensure that the final generated rotation matrix M r It can be more transformable and anti-interference. Then the global feature F g With F noise Perform feature splicing operation to obtain features For details, see formula (4):

[0042] F gn1 =concat(F g ,F noise ) (4)

[0043] Step 4: Get the rotation and scale transformation matrix M r . For the spliced ​​features Use (MLPs)2 for feature extraction to obtain a four-dimensional vector V = (w, x, y, z). Among them, (MLPs)2 includes a 512-channel one-dimensional convolution, a 256-channel one-dimensional convolution, and a 4-channel one-dimensional convolution. The four-dimensional vector is converted into the final matrix according to the correspondence between quaternions and rotation matrices. This process can be expressed as formulas (5), (6), and (7):

[0044]

[0045]

[0046]

[0047] like Figure 2 As shown, the jitter matrix M j For each point in the point cloud, jitter is performed, and the generation steps are as follows:

[0048] Step 1: Repeat stacking and concatenating the operation features. Different from the rotation matrix M r It is a transformation for a single point cloud, and the jitter matrix M j It is a transformation for each point in the point cloud, which requires a richer feature source. Therefore, here we stack and splice multiple levels of point cloud features and noise. g and randomly generated Gaussian noise F noise Repeat the stacking process n times, where n is 1024. Then, stack the F g and F noise Concatenate with the original high-dimensional feature F to obtain the concatenated feature For details, see formula (8):

[0049]

[0050] Step 2: Get the jitter matrix M j For the feature Use (MLPs)3 to perform feature extraction and generate the final jitter matrix Among them, (MLPs) 3 includes a 1024-channel one-dimensional convolution, a 128-channel one-dimensional convolution, a 64-channel one-dimensional convolution, and a 3-channel one-dimensional convolution, generating a jitter matrix M j The process can be expressed as formula (9) and (10):

[0051]

[0052]

[0053] For local data enhancement of point clouds, experimental verification shows that a mask matrix can also be obtained by learning a single point in the point cloud. However, if it is applied to the transformation of the original point cloud P, the transformed point cloud P t It is completely different from the original point cloud P, that is, it completely loses shape recognition and the model almost collapses. Therefore, this scheme starts from the local neighborhood, divides the point cloud into smaller areas, and then masks the point cloud according to the learned importance to obtain the mask matrix M m .

[0054] like Figure 2 As shown, the mask matrix M m The generation steps are as follows:

[0055] Step 1: Divide the point cloud into local neighborhoods. Use Farthest Point Sampling (FPS) to obtain the center points of m neighborhoods, and then use the K Nearest Neighbor (KNN) algorithm to obtain k points from the m center points, that is, divide the point cloud into m neighborhoods, each neighborhood includes k points. For specific implementation details, see formulas (11) and (12):

[0056] C m =FPS(P,m) (11)

[0057] P n =KNN(P,C m ,k) (12)

[0058] Among them, m represents the number of neighbors, represents m center points, k represents the number of points contained in the neighborhood, so the final result is

[0059] Step 2: Extract local neighborhood features. By using (MLPs)4, the point cloud P divided according to the local neighborhood is n Perform feature extraction to obtain feature information of several local areas (MLPs)4 includes two 64-channel 1D convolutions, one 128-channel 1D convolution, and one 1024-channel 1D convolution. The detailed process can be expressed as formulas (13) and (14):

[0060] F n =(MLPs)4(P n ) (13)

[0061]

[0062] Step 3: Get M neighborhood weight vectors F m . The vector F of the importance of M neighborhoods is obtained by global pooling m The specific global pooling operation includes using the maximum pooling operation on the high-dimensional features of a single point in the neighborhood and using the average pooling operation on all points in each neighborhood. For details, see formula (15):

[0063]

[0064] Step 4: Generate mask matrix M m . Design a Filter function that transforms the feature vector F composed of m neighborhoods m Each component in is sorted from low to high in importance. Then initialize It is a matrix of all 1s, and all the neighborhood index positions represented by the unimportant components are set to 0, and the final local mask matrix M is obtained. m . See formula (16):

[0065] M m =Filter(F m ) (16)

[0066] In this way, the global transformation matrix M is finally generated r and M j , and the local transformation matrix M m After that, the LPA module only needs to combine the original point cloud P and M r Do matrix multiplication and add the result to M j Do matrix addition and then add it to M m Point multiplication can get the final transformed point cloud P t , as shown in formula (17):

[0067]

[0068] in, represents matrix multiplication, represents matrix addition and e represents element-wise multiplication.

[0069] The purpose of step S1-2 is to achieve point cloud self-learning based on contrastive learning. Specifically, based on the original point cloud P and the newly generated point cloud P t , respectively, using weight-sharing encoders to extract features, thereby obtaining features F and F t , see the detailed process Figure 1 (a) After the FFP module, several layers of multi-layer perceptrons (MLPs) are added as projection layers. Then, F and F t Perform the self-learning process of the point cloud: Use the FFP module to output F pred and And used for the aforementioned input features F and F t prediction.

[0070] In the process of contrastive learning, it is hoped that the model can output similar features for samples of the same category and output features with large differences for samples of different categories. Model collapse means that the model outputs the same features for all input samples, regardless of whether they belong to the same category. In order to prevent the model from collapsing during contrastive learning, there are usually the following solutions: (1) construct a series of negative sample pairs and impose a large penalty on the judgment errors of negative samples; (2) explicitly learn several cluster centers and then divide the samples into different categories; (3) do not use positive and negative sample pairs, but use momentum encoders, gradient stop and other methods to further convert the contrastive learning problem into a prediction problem. Obviously, by constructing positive and negative sample pairs and clustering methods, a series of complicated processing steps are added. Therefore, the self-learning based on a single point cloud feature in this method adopts method (3), which requires fewer processing steps and the whole process is simpler.

[0071] Although P and P t They represent the same semantic category, but in general, after the LPA module transformation, P and P t There is a big difference, so the features F and F extracted by the encoder and the subsequent MLP projection layer are t Not the same either.

[0072] like Figure 3 As shown in the schematic diagram of the self-learning process of a single point cloud feature, F is output through the FFP module F pred , and then compare it with the feature F t The similarity measurement is to calculate the loss through the loss function l2. It is worth noting that in order to prevent the model from collapsing, F t The update of the gradient flow is also partially stopped, which on the one hand avoids the collapse of the model during the self-learning process, and on the other hand makes the learning of the network simpler and easier.

[0073] The original F and F t However, after F is input into the FFP module and continuously iterated learning, the obtained F pred With F t The difference between the features of F is getting smaller and smaller. This actually means that the model is learning to separate the features of F from those of F. t The parts with the same feature information are retained, and those with large differences are discarded. Also for F t The parts of F that have the same feature information as F are retained, and the feature information with large differences is discarded. In fact, the commonalities between similar point cloud samples are learned through the FFP module. These common features can be regarded as key feature information unique to each type of point cloud.

[0074] The ability of point clouds to learn unique feature information belonging to each category during self-learning is mainly due to the FFP module. It combines the attention mechanism to make fine-grained feature predictions on the original features, making the generated F pred and As close as possible to the original features F and F t Similar to the above, the prediction of features is completed. The structure of the FFP module is as follows Figure 4 As shown, to generate F from F pred Take this as an example, the specific steps are as follows:

[0075] Step 1: Nonlinear transformation feature F. The original feature F is transformed using nonlinear feature transformation to generate feature F a The nonlinear transformation here mainly includes two fully connected layers, a BatchNorm layer, and a ReLU layer, see formula (18):

[0076] F a =FC(σ relu (BN(FC(F)))) (18)

[0077] Among them, the input and output dimensions of the fully connected layer remain unchanged, so F a The dimension of also depends on the output dimension of the encoder used.

[0078] Step 2: Self-attention is used to obtain key features. After nonlinear transformation of the original feature F, feature F is obtained. a , and then use the self-attention mechanism to integrate key information. a Through the mapping of three fully connected layers, three weight matrices Q, K, and V are generated to fuse the final feature F pred The specific process can be expressed as formulas (19) and (20):

[0079]

[0080]

[0081] Among them, d F Represents the dimension of feature F.

[0082] The above two steps, namely nonlinear transformation and self-attention fusion, are mainly used to t The same feature information is better learned and retained. This enables the entire model to output similar features for point clouds of the same category, while point clouds of different categories output features with greater differences, thus having better feature discrimination.

[0083] The adaptive loss function includes the loss function l1 optimized for data enhancement and the loss function l2 optimized for the point cloud self-learning process. Its adaptability is reflected in the dynamic trade-off between the loss function l1 and the loss function l2, that is, achieving a balance between the two stages of data enhancement and point cloud self-learning, so that the final loss function value is minimized.

[0084] The purpose of designing the loss function l1 is to use the original point cloud P as the standard to generate a point cloud P with as large a difference as possible. t , that is, to improve the data enhancement effect. The purpose of designing the l2 loss function is to achieve the similarity prediction between the two point cloud features, that is, to better complete the self-learning process of the point cloud. Obviously, if the LPA module over-enhances the original point cloud, the difficulty of predicting feature similarity will become greater, that is, the value of the loss function l1 becomes smaller, but at the same time the value of the loss function l2 becomes larger. Similarly, if the LPA module only performs simple data enhancement, it will inevitably cause the similarity to be too high, but the feature similarity prediction becomes easier, that is, the value of the loss function l2 becomes smaller, but at the same time the value of the loss function l1 becomes larger. Therefore, by continuously optimizing the loss function, we try to find a loss value that makes both the loss function l1 and the loss function l2 relatively small, so that the model can eventually achieve a better performance.

[0085] Regarding the construction of the loss function l1: It is hoped that even if the original point cloud P and the newly generated point cloud P t There are large differences between them, but as long as they represent the same category at the semantic level, they can learn the same feature information in the subsequent point cloud self-learning process. Obviously, if the LPA module simply takes the original point cloud P as P t , that is, the LPA module collapses and does not perform any learning. Then in the process of point cloud self-learning, two identical point clouds must learn the same features, but in fact the entire model does not learn any useful information. Therefore, in the process of designing the LPA module loss function l1, it is best to make P and P t There is a big difference. This scheme constructs the similarity between point clouds through cosine similarity instead of calculating the Euclidean distance between point clouds. The reason is that the calculation of Euclidean distance is more susceptible to the influence of outliers and outliers in the point cloud. For this reason, it is often necessary to normalize the point cloud. The process of calculating cosine similarity already includes the L2 normalization of the original point cloud, see formula (21):

[0086]

[0087] In general, the smaller the loss function, the better the optimization of model parameters. However, here, since cos(P,P t)∈[-1,1], if the loss function becomes the minimum value, that is, -1, it only represents P and P t Completely negative correlation. Only when cos(P,P t )=0, from the perspective of vectors, P and P t When they are vertical, the two are completely unrelated. Therefore, the loss function needs to be designed to perform a gradient descent of the minimum value in the direction of 0, and the loss function is redesigned as formula (22):

[0088]

[0089] in, is a symbolic function, which represents the relationship between P and P t The cosine similarity takes the sign value. In this way, when the cosine similarity is negative, since the sign function returns -1, l1 is still a positive value. This ensures that the loss function continues to converge towards the value of 0 during the training process, and generates a point cloud P that is as different from the original point cloud P as possible. t .

[0090] Regarding the construction of the loss function l2: For P and P with large differences t , the features F and F extracted by the encoder t There should be a big difference, but due to the FFP module generated by pred and It is possible to transform the original features F and F t To predict features, we have F pred ≈F t , Here, cosine similarity can also be used to measure the difference between two features, as shown in formula (23):

[0091]

[0092] Among them, l2∈[-1,1], when the features can completely predict each other, l2=-1, which is the most ideal value state.

[0093] Therefore, the two sub-loss functions, loss function l1 and loss function l2, are combined to construct the adaptive loss function in this method. In the actual experimental process, it is found that adding a hyperparameter of weight α to l1 has a better effect, so the final adaptive loss function is shown in formula (24):

[0094] l=αl1+l2, l∈[-1,1+α] (24)

[0095] The PointSL model and its training method are analyzed through experiments below.

[0096] Datasets: ShapeNetCore, ModelNet40, and ShapeNetPart are used to pre-train and verify the superiority and advancement of the algorithm model. The ShapeNetCore dataset is used for self-supervised representation learning of the PointSL model and does not use any point cloud labels on the dataset. ModelNet40 and ShapeNetPart are used for downstream point cloud representation learning tasks. Among them, the ModelNet40 dataset is mainly used for performance comparison of point cloud classification tasks, and ShapeNetPart is mainly used for performance comparison of point cloud part segmentation tasks. The ModelNet dataset is an open source 3D point cloud dataset provided by Princeton University. As its sub-dataset, ModelNet40 contains 40 categories of objects including airplanes, bathtubs, beds, and stools, with a total of 12,311 samples (9,843 in the training set and 2,468 in the validation set). ShapeNet is a large-scale point cloud dataset with rich annotations, of which ShapeNetCore and ShapeNetPart are two sub-datasets. Among them, ShapeNetCore contains 55 common object categories and a total of 51,300 specific point cloud samples. ShapeNetPart is a representative of point cloud part segmentation datasets, which includes 16 object categories, 50 part categories, and a total of 16,881 point cloud samples (14,007 in the training set and 2,874 in the test set).

[0097] Environment: The hardware environment uses Nvidia Titian Xp dual graphics cards with a video memory size of 24GB, the CPU model is Intel Xeon Silver 4116 processor, the memory size is 64GB, the operating system uses Ubuntu 20.04LTS version, Python version is 3.6.13, Pytorch version is 1.10.0, and cuda version is 10.2.

[0098] Training parameter configuration: The unlabeled ShapeNetCore dataset is used to train the PointSL model, and the batch size is set to 32. The learning rate decay method of cosine annealing is used, and the initial learning rate is set to 0.01. After training the PointSL model for 100 epochs, the encoder is stripped out and used in representation learning tasks such as point cloud classification and segmentation. The PointSL model is trained in a self-supervised manner on the ShapeNetCore dataset without using any label information, and a more generalized model is obtained for subsequent transfer learning. For the classification and segmentation tasks, only a small learning rate is used to fine-tune the encoder, and the learning rate of the encoder is set to 0.0005. However, the fully connected layer of the classification task and the decoder of the segmentation task are still trained with a larger learning rate, because they are added to the model according to the needs of downstream tasks and have not been pre-trained before. The learning rate of these layers is set to 0.005. The entire model training process uses a stochastic gradient descent optimizer, and the momentum is set to 0.9. Weight decay is used to avoid overfitting of the network, and the decay coefficient is 0.05.

[0099] Evaluation indicators: For classification tasks, OA (Overall Accuracy) and AA (Average Accuracy) are used to evaluate the performance of the PointSL model. The former is calculated by calculating the percentage of all samples correctly classified to the total number of samples. The latter is calculated by first calculating the percentage of each class of samples correctly classified to the total number of samples in each class, and then averaging the accuracy of each class of samples. For details, see formulas (25) and (26):

[0100]

[0101]

[0102] Among them, s i Represents the correctness of the classification result. If the classification is correct, then s i =1, if the classification is wrong, then s i = 0. n represents the total number of point cloud samples, c i It represents the classification accuracy of each type of samples, m is the number of categories, and k is the number of point cloud samples contained in each category.

[0103] For some segmentation tasks, the intersection over union (IoU) is used to evaluate the performance of the PointSL model. IoU is calculated by the ratio of the intersection and union of the actual segmentation area of ​​the point cloud and the segmentation area inferred by the model, see formula (27):

[0104]

[0105] Among them, A gt A represents the pre-annotated point cloud segmentation area ground truth, p Represents the point cloud segmentation area predicted by the model. The average intersection-over-union is the average IoU of a given number of samples. It is mainly divided into instance average intersection-over-union and category average intersection-over-union. The instance average intersection-over-union (Inst.mIoU) is the average IoU of all samples, and the category average intersection-over-union (Cls.mIoU) is to first calculate the average IoU of each category of point cloud samples, and then sum and average these results. As shown in formulas (28) and (29):

[0106]

[0107]

[0108] Among them, s i(IoU) Represents the IoU value of a single point cloud, c i(IoU) Represents the average IoU value of the i-th category point cloud, n is the total number of samples, m is the number of categories, and k represents the number of samples in each category.

[0109] The following introduces the performance comparison of the classification task, using ModelNet40 as the point cloud dataset for the classification task.

[0110] 1. Compare the classification performance with similar supervised methods.

[0111] Figure 5 This is a schematic diagram comparing the classification accuracy obtained by using traditional supervised model training and PointSL model training on ModelNet40.

[0112] Table 1. Comparison with the corresponding supervised methods OA and AA on ModelNet40

[0113]

[0114] pass Figure 5The experiments shown in Table 1 show that the performance of this method has surpassed traditional supervised methods. Table 1 compares the results of training four typical supervised models PointNet, PointNet++, DGCNN, and PointMLP on the ModelNet40 dataset. Here, the OA and AA obtained by training with traditional supervised methods are compared, as well as the OA and AA obtained by self-supervised training with the PointSL model and transfer learning for downstream classification tasks. As can be seen from Table 1, the OA and AA of this method on PointNet are 90.7% and 87.9%, which are 1.5% and 1.7% higher than the supervised methods, respectively. The OA and AA on PointNet++ are 92.4% and 89.4%, which are 1.7% and 1.6% higher, respectively. The OA and AA on DGCNN are 93.3% and 90.8%, which are 1.0% and 0.9% higher, respectively. The OA and AA on PointMLP are 94.2% and 91.4%, which are 0.9% and 0.8% higher, respectively.

[0115] The reason for this is attributed to the point cloud self-learning step based on contrastive learning. This step first pre-trains a more generalized encoder model on the ShapeNetCore dataset, and then performs transfer learning on the downstream ModelNet40 point cloud classification dataset. Obviously, compared with training directly on the ModelNet40 dataset, it can produce better classification performance. Moreover, the designed LPA module can perform specific data transformations from both global and local dimensions for each original point cloud. Compared with the traditional random point cloud enhancement method, the enhancement effect of this method is obviously better. This better enhancement effect can make the original point cloud and the transformed point cloud have a greater difference, which makes the model representation learning process more challenging. In addition, in the contrastive learning process of the point cloud, by constructing the FFP module, it combines the attention mechanism to better predict the mutual prediction between features, so that similar point cloud samples can learn similar features as much as possible, and the feature differences between different point cloud samples are as large as possible, so the PointSL model finally trained has better feature recognition. Finally, the designed adaptive loss function can strike a balance between data enhancement and point cloud self-learning, preventing the model from collapsing in the representation learning process. Therefore, compared with traditional supervised methods, this method can obtain an encoder model with better performance and stronger robustness, and after fine-tuning on the subsequent ModelNet40 point cloud dataset, it can obtain better classification results than supervised methods.

[0116] 2. Comparison of classification performance of point clouds with few samples and high structural complexity.

[0117] Table 2. Comparison of point cloud accuracy AA between traditional supervised training and PointSL self-supervised training methods

[0118]

[0119]

[0120] From Table 2, it can be concluded that for the case where the number of point cloud samples is small and the structure complexity of the point cloud itself is high, the encoder trained using this method has better classification accuracy. Here, taking the PointNet++ model as an example, the 10 most typical categories on ModelNet40 are selected, and the AA indicators of the supervised PointNet++ and the PointNet++ model trained by the PointSL self-supervised method are compared. As can be seen from Table 2, for point cloud samples such as airplane, keyboard, and laptop, PointNet++ and PointNet++ trained using the PointSL model can achieve 100% classification accuracy. For bed and desk, both can also achieve high accuracy. However, on point clouds such as dresser, plant, radio, and stool, the classification effect is not very ideal. However, on the PointNet++ model trained in a self-supervised manner using the PointSL model, the overall effect is better than the PointNet++ model trained in a traditional supervised manner, and its accuracy on dresser, plant, radio, and stool has increased by 5%, 1.7%, 2.9%, and 4.4%, respectively.

[0121] Obviously, the number of point cloud samples and the complexity of the object represented by the point cloud have a profound impact on the accuracy of classification. For point clouds with relatively simple structures, such as bed, desk, keyboard, laptop and other categories (mostly cubic structures), the encoders obtained by the two training methods can easily learn the effective feature information, and the classification accuracy is very high. Although the structure of airplane is more complex than the aforementioned point clouds, the classification effect is also better due to the large number of samples. The most noteworthy are the point clouds with few samples (stool) and the point clouds with high structural complexity (dresser, plant, radio). This method can perform better than the supervised training method. This is because the PointSL model is used for point cloud self-learning based on contrastive learning, and it is used for transfer learning in downstream classification tasks. Compared with traditional supervised methods for specific point cloud datasets, the encoder trained by this method has better generalization, so it can also obtain higher accuracy on point cloud categories with few samples and high structural complexity. Traditional supervised learning methods only learn for a single dataset, which is obviously prone to overfitting of the model. Among them, compared with the traditional random data enhancement, the LPA module can obviously make the encoder learning process more challenging, thereby enhancing the robustness of the encoder. By applying the FFP module for comparative learning in the process of point cloud self-learning, it makes it easier for the encoder to identify features of different categories. Therefore, compared with the traditional supervised learning method, this method is easier to improve the recognition of features, and then dynamically balances the two through an adaptive loss function, ultimately enabling point cloud samples with more complex structures and point cloud samples with smaller data sizes to obtain higher category accuracy.

[0122] 3. Comparison of classification performance of point clouds with high feature similarity.

[0123] Figure 6 The figure is a comparative diagram of the visualization of the confusion matrix of traditional supervised training and training using the PointSL model, where (a) is a visualization of the confusion matrix of traditional supervised training, and (b) is a visualization of the confusion matrix of training using the PointSL model.

[0124] Figure 6 It shows that for point cloud samples with high feature similarity, the encoder trained by this method has higher classification accuracy, that is, it has better feature discrimination. Here, taking the DGCNN network as an example, the confusion matrix of the models trained by two different methods on ModelNet40 is shown. Figure 6(a) is a visualization of the confusion matrix of the model obtained by DGCNN using supervised training on ModelNet40. The OA and AA of this model are 92.3% and 89.9% respectively, and the number of correctly judged samples is 2277. Figure 6 (b) is a visualization of the confusion matrix of the model obtained by this method on ModelNet40. The OA and AA of this model are 93.3% and 90.8% respectively, and the number of correctly judged samples is 2302. The latter is 1.0% and 0.9% higher than the former in OA and AA respectively, and the number of correctly judged samples is 25 more. By comparing the confusion matrix to reveal the improvement of the classification accuracy of the original DGCNN encoder using the PointSL unsupervised training method, it can be clearly noticed that the number of point cloud samples in the night_stand category that are mistakenly judged as dresser has decreased from 15 to 11, a decrease of 4, the number of point cloud samples in the plant category that are mistakenly judged as flower_pot has decreased from 19 to 15, a decrease of 4, and the number of point cloud samples in the table category that are mistakenly judged as desk has decreased from 21 to 16, a decrease of 6. The number of point clouds in other categories such as bathtub, bookshelf, tv_stand, etc. that are mistakenly judged as similar categories has decreased. And these point cloud category models are extremely high in similarity. Therefore, the number of samples that the model misjudged for these categories is also the highest among all categories. This proves to a certain extent that this method can have better feature discrimination for different category models, and even for point cloud categories with high similarity, it has a high model discrimination ability.

[0125] The reason why this method can have better feature discrimination, especially on point clouds with high similarity in these categories, is that unlike the previous supervised learning training method, this method uses a point cloud self-learning method based on contrastive learning, which is easier to perform transfer learning through the integration of various modules, thereby obtaining a more generalized and robust encoder. Here, in order to get rid of the limited effect of traditional random data enhancement, the LPA module is used. In the process of contrastive learning, unlike the previous method of constructing complex positive and negative sample pairs, the FFP module and gradient stop strategy are used to predict each other between features. This method makes it easier for the encoder to learn the commonality (key information) between two point cloud samples and discard the difference information (unimportant information) between the two. After learning from a large number of point cloud samples, the encoder can naturally improve its ability to identify features of point clouds of different categories. In addition, the adaptive loss function is also constantly optimizing the training process of the network model. All of these combined make the model trained according to this method have better feature discrimination, and even on point clouds with high similarity, a high judgment accuracy can be obtained.

[0126] The following introduces the performance comparison of the part segmentation task, using ShapeNetPart as the point cloud dataset for the part segmentation task.

[0127] 1. Compare the segmentation performance with similar supervised methods.

[0128] Table 3. Comparison of Inst.mIoU and Cls.mIoU with corresponding supervised methods on ShapeNetPart

[0129]

[0130] Through the experiments in Table 3, it is verified that the encoder trained by this method is also better than the traditional supervised training method in various metrics on part segmentation. Here, the point cloud part segmentation dataset ShapeNetPart is used to compare the segmentation performance indicators of the four models PointNet, PointNet++, DGCNN, and PointMLP. These models are trained using traditional supervised methods and PointSL self-supervised methods, and their Inst.mIoU and Cls.mIoU evaluation indicators are displayed. As can be seen from Table 3, the Inst.mIoU and Cls.mIoU metrics of this method on PointNet are 84.5% and 81.2%, respectively, which are 0.8% and 0.8% higher than the supervised training method. The Inst.mIoU and Cls.mIoU metrics on PointNet++ are 85.5% and 82.5%, which are 0.4% and 0.6% higher than the supervised training method. The metrics of Inst.mIoU and Cls.mIoU on DGCNN are 85.8% and 82.6%, which are 0.6% and 0.3% higher than those of supervised training methods, respectively. The metrics of Inst.mIoU and Cls.mIoU on PointMLP are 86.3% and 85.1%, which are 0.4% and 0.5% higher than those of supervised training methods, respectively.

[0131] Through the above comparison, it is not difficult to find that the use of this method to train these models has also improved the metric indicators of the models in the point cloud partial segmentation task, but the improvement is significantly inferior to the point cloud classification task. Further analysis of the reason is that the PointSL model only pre-trains the encoder part, thereby obtaining an encoder model with superior performance, better generalization and robustness. However, for the many upsampling layers of the partial segmentation task, that is, the decoder part, it does not participate in the pre-training with the self-supervised training method of the PointSL model, and transfer learning to the downstream task, which affects the final segmentation accuracy and makes the improvement of the segmentation mIoU limited. However, compared with the traditional supervised method, its segmentation performance has been further improved. This also proves that the superior performance, better generalization and robustness of the encoder model also help to improve the segmentation representation learning ability to a certain extent. The mutual cooperation of the LPA module, the FFP module, and the adaptive loss function in the point cloud self-learning process enables this method to obtain an encoder model with better performance and better generalization, and then at the feature extraction level, it has better feature recognition for different category areas of a single sample. In subsequent segmentation tasks, these similar feature areas will be more easily upsampled into the same segmentation areas as the ground truth, thereby improving the segmentation accuracy.

[0132] 2. Comparison of segmentation performance on point clouds with few samples and high structural complexity.

[0133] Table 4. Comparison of Cls.mIoU (%) between traditional supervised methods and PointSL unsupervised methods

[0134]

[0135] The experimental results in Table 4 show that for point cloud category samples with few samples and high structural complexity, the segmentation performance indicators of the model trained by this method are also better. Taking the encoder of PointMLP as an example, the average intersection-over-union ratio of each category on the ShapeNetPart dataset is analyzed in detail. Here, the traditional supervised training method and the unsupervised training method using this PointSL are compared respectively. As can be seen from Table 4, the Inst.mIoU and Cls.mIoU of PointSL on the ShapeNetPart dataset are 86.3% and 85.1%, respectively, which are 0.4% and 0.5% higher than the original supervised training method. Further observing the average intersection-over-union ratio of the 16 point cloud categories, it can be seen that the point cloud categories with high segmentation accuracy in both methods include Cap, Chair, Guitar, Laptop, Mug, etc. The average intersection-over-union ratios of these categories are all above 90%. This is mainly because the objects represented by these point clouds themselves have a relatively simple structure or the number of samples of this category of point clouds is large. For example, the point cloud of the Aeroplane category has a relatively complex structure, including the fuselage, wings and wheels, but due to the large number of samples, the overall segmentation accuracy is relatively good. For Laptop, the structure itself is relatively simple, and it is easier for the model to learn features for segmentation. It is worth noting that for point clouds of categories such as Car, Ear-Phone, Motor-Bike and Rocket, the segmentation accuracy of the two methods is relatively low, and the average intersection-over-union ratio of the categories is less than 80%. Obviously, this is because the point cloud samples of these categories are subject to the complex structure of the object itself or the small number of samples, so the segmentation accuracy is low. However, this method is still better than the traditional supervised training method. The average segmentation accuracy of this method on the point cloud categories of Car, Ear-Phone, Motor-Bike and Rocket is 80.1%, 82.4%, 79.2%, and 66.1%, which are 0.3%, 3.5%, 3.1% and 1.5% higher than the original supervised training method, respectively.

[0136] Further analysis shows that this method uses a pre-training step to obtain a model with better generalization and robustness. Applying this model to transfer learning of point cloud segmentation can improve the discriminability of different features. Therefore, fine-tuning training on the ShapeNetPart dataset produces better performance indicators than supervised methods. Obviously, a model with better generalization and robustness is more likely to resist the adverse effects of factors such as the small number of samples in the segmentation dataset and the high structural complexity of the model itself.

[0137] 3. Visualization of partial segmentation on point cloud samples with few samples and high structural complexity.

[0138] In order to further verify that the model obtained by this method has better segmentation performance on point cloud categories with few samples and high structural complexity, a detailed segmentation visualization is performed on the ShapeNetPart dataset for some of the most representative point cloud categories. Figure 7 and Figure 8 As shown in Figure 2, the segmentation effect on PointMLP after training the PointMLP model using the traditional supervised method and this method is further compared. As shown in Table 4 above, the Cls.mIoU of the two training methods is relatively high on point clouds of categories that are easier to segment, such as Laptop, Mug, and Guitar. Figure 6 The visualization results also prove this point. Whether it is the traditional supervised method or the method used to train PointMLP, the overall segmentation effect is almost the same as the ground truth. However, when segmenting difficult category point cloud samples, the model trained by this method is better than the traditional supervised method in local segmentation details. Figure 8 It can be clearly seen that this method is significantly better than the traditional supervised method in the point clouds of the aeroplane, earphone and motorbike categories. In particular, for the segmentation of the earphone, it can be seen that the supervised training PointMLP is almost completely wrong in the segmentation of the headband part of the earphone, while this method can correctly segment the headband part adjacent to the two ear cushions. In addition, this method also performs better in the segmentation of the nose and wings of the rocket and the axle of the motobike.

[0139] 5. Comparison of convergence of loss functions.

[0140] Compared with the traditional random enhancement method, the LPA module proposed in this scheme converges more smoothly in the loss function and has a higher overall convergence amplitude. Fig. 9 It further demonstrates the convergence of the model loss function of point cloud samples using different data augmentation methods, namely traditional random augmentation and using the LPA module on four encoder models PointNet, PointNet++, DGCNN, and PointMLP. Since the adaptive loss function includes the loss function l1 optimized for the LPA module and the loss function l2 optimized for the FFP module. Therefore, when the traditional random augmentation method is applied to train the PointSL model, the adaptive loss function only includes the loss function l2, whose main function is to predict and optimize the features of the two point clouds extracted by the encoder. Fig. 9It can be seen that when the random enhancement method is applied to the point cloud samples on the four encoder models, the value of the loss function is very small at the beginning, and its value is less than -0.6 (measured by cosine similarity, when the loss function is -1, it means that the two features are exactly the same). This proves that the effect of random enhancement is very limited, and the similarity between the transformed point cloud and the original point cloud is very high. Therefore, after the model uses random enhancement, the loss function converges quickly, but the encoder itself does not perform a lot of effective learning and weight adjustment. On the other hand, when using the LPA module, the loss function value is relatively large at the beginning. As the number of iterations increases, the loss function continues to converge, and the encoder undergoes a gradual learning process. At the same time, it can also be observed that the jitter of the entire loss function is more obvious through the traditional random enhancement method. Since the traditional random enhancement has no gradient feedback mechanism, the random enhancement effect may be better in some cases, while the random enhancement effect is more limited in other cases. This will affect the performance of the point cloud in the self-supervised learning process, and then affect the accuracy of the model in downstream representation learning tasks.

[0141] 6. Ablation experiment.

[0142] The FFP module in this PointSL model is the basic module that constitutes contrastive learning. As a self-supervised method, it has been compared and analyzed in detail with supervised methods in all previous experiments. Therefore, in the ablation experiment part, we focus on analyzing and discussing the LPA module and comparing it with the random data enhancement method.

[0143] Table 5. Ablation experiment of LPA module (a) Comparison of classification performance on ModelNet40

[0144]

[0145] (b) Comparison of segmentation performance on ShapeNetPart

[0146]

[0147] Table 5 takes the PointNet model as an example to show the effectiveness of the LPA module in point cloud classification and segmentation. Whether it is the global point cloud transformation part or the local point cloud transformation part in the LPA module, the final classification and segmentation performance indicators are improved. From Table 5(a), it can be seen that on the ModelNet40 dataset, compared with the traditional random data enhancement, after using the global data enhancement of the LPA module, its OA and AA are improved by 0.6% and 0.6% compared with the random data enhancement method. After using the local data enhancement of the LPA module, its OA and AA are improved by 0.3% and 0.5% compared with the random data enhancement method. When the LPA module is fully applied, its OA and AA are improved by 1.1% and 1.4% compared with the random enhancement method. From Table 5(b), it can be seen that on the ShapeNetPart dataset, compared with the traditional data enhancement, after using the global data enhancement of the LPA module, its Inst.mIoU and Cls.mIoU are improved by 0.4% and 0.2% respectively. After local data augmentation using the LPA module, its Inst.mIoU improves by 0.2% and 0.1%. When the LPA module is fully applied, its Inst.mIoU and Cls.mIoU improve by 0.7% and 0.5%, respectively.

[0148] Both the global data enhancement and the local data enhancement in the LPA module have a positive impact on the final classification and segmentation performance indicators. It performs progressive enhancement from both global and local dimensions in a learnable way, and has a gradient feedback mechanism. Compared with traditional random enhancement methods, its enhancement effect on specific point cloud samples is significantly better, making the point cloud self-supervised learning process more challenging. This is conducive to training a model with better generalization and robustness, and then applying it to downstream tasks in the form of transfer learning, thereby continuously improving the performance of point cloud representation learning.

[0149] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. A point cloud self-supervised learning network model based on contrastive learning, characterized in that: include: A learnable point cloud enhancement module is used to perform global and local transformations in a learnable manner according to different categories of point cloud samples to achieve data enhancement and optimization; A refined feature prediction module is used to combine the attention mechanism to perform feature mutual prediction between point cloud sample pairs, thereby improving the feature discrimination of the network model for different categories of point clouds; as well as The adaptive loss function is used to continuously optimize the network model through a gradient feedback mechanism to avoid model collapse.

2. The point cloud self-supervised learning network model based on contrastive learning according to claim 1, characterized in that: in, Define the point cloud to be enhanced and optimized as the original point cloud P, The learnable point cloud enhancement module has the following functions: For global data enhancement, generate the rotation and scale transformation matrix M r and the jitter matrix M j , For local data enhancement, generate a mask matrix M m , Combine the original point cloud P with the rotation and scale transformation matrix M r , jitter matrix M j , mask matrix M m Perform transformation operations to generate the final point cloud P t .

3. The point cloud self-supervised learning network model based on contrastive learning according to claim 2, characterized in that: in, The rotation and scaling transformation matrix M r The generation of includes the following steps: Step 1: Obtain high-dimensional features F h :For the original point cloud P, use (MLPs)1 to extract its features and obtain the feature F, which is represented by formulas (1) and (2). Among them, (MLPs) 1 includes two 64-channel one-dimensional convolutions, one 128-channel one-dimensional convolution, and one 1024-channel one-dimensional convolution, and the one-dimensional convolution layers at each level include BN layers and activation functions ReLU. p i Represents any point in the original point cloud P, MLP j Including multiple convolutional layers, BatchNorm layers and activation function layers, c in represents the number of input channels, c out Represents the number of output channels, σ represents the nonlinear activation function layer and the ReLU function is used here; Step 2: Generate global feature F g :For high-dimensional features F h Use max pooling to get global features It is represented by formula (3), F g =maxpool(F h ) (3); Step 3: Feature concatenation operation: Generate random noise features using Gaussian distribution The global feature F g With F noise Perform feature splicing operation to obtain features It is represented by formula (4), Step 4: Get the rotation and scale transformation matrix M r :For the features after splicing Use multi-layer perceptrons (MLPs)2 to extract features and obtain a four-dimensional vector V = (w, x, y, z), which is converted into the final matrix according to the correspondence between quaternions and rotation matrices. It is represented by formulas (5), (6), and (7), Among them, (MLPs)2 includes a 512-channel one-dimensional convolution, a 256-channel one-dimensional convolution, and a 4-channel one-dimensional convolution.

4. The point cloud self-supervised learning network model based on contrastive learning according to claim 3, characterized in that: in, The jitter matrix M j The generation of includes the following steps: Step 1: Repeat stacking and concatenating operation features: transform the global feature F g and randomly generated Gaussian noise F noise Repeat the stacking n times, and then stack the F g and F noise Concatenate with the original high-dimensional feature F to obtain the concatenated feature It is represented by formula (8), Step 2: Get the jitter matrix M j : For the feature Use (MLPs)3 for feature extraction and generate the final jitter matrix It is represented by formula (9) and (10), c in {2112,1024,128,64},c out [{1024,128,64,3},σ=ReLU(g)(10) Among them, (MLPs) 3 includes a 1024-channel one-dimensional convolution, a 128-channel one-dimensional convolution, a 64-channel one-dimensional convolution, and a 3-channel one-dimensional convolution.

5. The point cloud self-supervised learning network model based on contrastive learning according to claim 2, characterized in that: in, The mask matrix M m The generation of includes the following steps: Step 1: Divide the point cloud into local neighborhoods: Use the farthest point sampling to obtain the center points of m neighborhoods, and then use the K nearest neighbor algorithm to obtain k points from the m center points, that is, divide the point cloud into m neighborhoods, each neighborhood includes k points, represented by formulas (11) and (12), C m =FPS(P,m) (11) P n =KNN(P,C m ,k) (12) Among them, m represents the number of neighbors, represents m center points, k represents the number of points contained in the neighborhood, and finally generates Step 2: Extract local neighborhood features: Use (MLPs) 4 pairs of point clouds P n Perform feature extraction to obtain feature information of several local areas It is represented by formula (13) and (14), F n =(MLPs)4(P n ) (13) Among them, (MLPs)4 includes two 64-channel 1D convolutions, one 128-channel 1D convolution, and one 1024-channel 1D convolution; Step 3: Get M neighborhood weight vectors F m :Get the vector F of M neighborhood importances through global pooling m , the global pooling operation includes using the maximum pooling operation on the high-dimensional features of a single point in the neighborhood and the average pooling operation on all points in each neighborhood, which is represented by formula (15), Step 4: Generate mask matrix M m :Design a Filter function that transforms the feature vector F composed of m neighborhoods m Each component in is sorted from low to high in importance, and then initialized It is a matrix of all 1s, and all the neighborhood index positions represented by the unimportant components are set to 0, and the final local mask matrix M is obtained. m , represented by formula (16), M m =Filter(F m ) (16).

6. The point cloud self-supervised learning network model based on contrastive learning according to claim 2, characterized in that: in, For the original point cloud P, rotation and scale transformation matrix M r , jitter matrix M j , mask matrix M m The transformation operation is: transform the original point cloud P and M r Do matrix multiplication and add the result to M j Do matrix addition and then add it to M m Point multiplication, get point cloud P t , represented by formula (17), in, represents matrix multiplication, represents matrix addition and e represents element-wise multiplication.

7. The point cloud self-supervised learning network model based on contrastive learning according to claim 1, characterized in that: in, Define the input feature of the refined feature prediction module as F and the output feature as F pred , The refined feature prediction module generates feature F according to feature F pred The following steps are involved: Step 1: Nonlinear transformation feature F: Use nonlinear feature transformation on feature F to generate feature F a , the nonlinear transformation includes two fully connected layers, a BatchNorm layer, and a ReLU layer, represented by formula (18), F a =FC(σ relu (BN(FC(F)))) (18) Among them, the input and output dimensions of the fully connected layer remain unchanged; Step 2: Self-attention to obtain key features: Use the self-attention mechanism to a Map three fully connected layers to generate three weight matrices Q, K, and V, which are fused to obtain the final feature F pred , represented by formula (19) and (20), Among them, d F Represents the dimension of feature F.

8. The point cloud self-supervised learning network model based on contrastive learning according to claim 1, characterized in that: in, The adaptive loss function includes a loss function l1 optimized for data enhancement and a loss function l2 optimized for the point cloud self-learning process. The loss function l1 is expressed as formula (22), l1=sign(cos(P,P t ))*(cos(P,P t )) (22) Among them, P represents the original point cloud, P t represents the point cloud after data enhancement and optimization by the learnable point cloud enhancement module according to the point cloud P, sign(g) is the sign function, which represents the t The cosine similarity of takes the symbol value, The loss function l2 is expressed as formula (23), Among them, F and F t represents the original feature, F pred and F tpred The refined feature prediction module is based on the features F and F t The output features, The adaptive loss function is expressed as formula (24), l＝αl1+l2, l∈[-1,1+α] (24) Among them, α represents the weight.

9. A training method for a point cloud self-supervised learning network model based on contrastive learning according to any one of claims 1 to 8, characterized in that: The following steps are involved: Step S1, pre-training the network model; Step S2, fine-tuning and testing the trained network model on downstream tasks; Wherein, in the step S1, the network model is continuously optimized by an adaptive loss function during the pre-training process, and the pre-training includes the following sub-steps: Step S1-1, using a learnable point cloud enhancement module to perform data enhancement on the original point cloud from both global and local perspectives in a learnable manner; Step S1-2: first use a weight-sharing encoder to extract features from the original point cloud and the transformed point cloud, and then use a refined feature prediction module to perform feature mutual prediction between the two point cloud samples to achieve self-learning of point cloud representation.

10. The training method according to claim 9, characterized in that: in, In the step S2, for the point cloud classification task, a fully connected layer is set to evaluate the performance index, and for the point cloud segmentation task, a decoder is set to evaluate the performance index.

Images