A 3D point cloud upsampling method based on attention mechanism

Abstract

The present invention discloses a three-dimensional point cloud upsampling method based on an attention mechanism, comprising the following steps: Step 1: preprocessing three-dimensional point cloud data; Step 2: constructing a three-dimensional point cloud upsampling network structure based on an attention mechanism; Step 3: establishing a loss function between a target point cloud and an output point cloud; Step 4: using a public data set for training and testing, and outputting the final result. The present method proposes a three-dimensional point cloud upsampling network based on an attention mechanism, further strengthens the learning of local features and neighborhood relationships, and adds a residual structure to optimize the information transfer process, thereby improving the accuracy of three-dimensional point cloud upsampling and reducing the loss between the target point cloud and the upsampled point cloud. The experimental results clearly confirm the efficiency and practicality of the proposed method, and the method of the present invention is more efficient and accurate for three-dimensional point cloud upsampling.

Description

Technical Field

[0001] The present invention relates to the field of upsampling of three-dimensional point clouds. Three-dimensional point clouds are the standard output form of sensors. How to process the sampled point cloud information has attracted more and more attention, so a three-dimensional point cloud upsampling method based on an attention mechanism is proposed. Background Art

[0002] With the rapid development of 3D acquisition technology, the cost of 3D sensors continues to decrease, and they are increasingly widely used in all aspects of production and life, including various types of 3D scanners, lidars, and RGB-D cameras (such as Kinect, RealSense, and Apple Depth Camera). The three-dimensional data obtained by these sensors can provide rich geometry, shape, and scale information. Combined with 2D images, 3D data can be used to better understand the surrounding environment. 3D data has many applications in different fields, including autonomous driving, robotics, remote sensing, and medical treatment.

[0003] Three-dimensional data can usually be represented in different formats, including depth images, point clouds, meshes, and volumetric meshes. Point cloud representation, as a common format, is a recently popular and widely used form of sensor output. It retains the original geometric information in three-dimensional space without any discretization. Therefore, it is the preferred representation for many scene understanding related applications, such as autonomous driving and robotics. However, there are several problems with the raw three-dimensional point cloud data obtained from the sensor: first, the point cloud is usually sparse, second, there is noise, and third, the distribution is non-uniform. Therefore, it is particularly important to design an algorithm to upsample the sparse point cloud to make it dense and uniform for further rendering, analysis, and processing.

[0004] In recent years, deep learning technology has dominated many research fields, such as computer vision, speech recognition, and natural language processing. Qi et al. first proposed PointNet and PointNet++ to directly process raw 3D point cloud information, and solved the disorder problem of point cloud by applying shared multi-layer perceptrons and symmetric functions (such as maximum pooling). On this basis, Yu et al. first proposed an end-to-end 3D point cloud upsampling network (PU-Net), which obtained the features from the encoder by copying n input decoders to obtain the upsampled point cloud. Wang Yifan et al. proposed MPU, which gradually obtained dense point clouds using a multi-step training method. Ruihui Li et al. proposed PU-GAN, which introduced a generative adversarial network to further guide the upsampling process of point clouds. Google first proposed the attention mechanism, the core of which is to adjust the allocation of attention to the input weights to optimize the network structure. Jie Hu et al. proposed a channel-based attention mechanism for the upsampling process of 2D images. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and provide a three-dimensional point cloud upsampling method based on an attention mechanism.

[0006] A three-dimensional point cloud upsampling method based on an attention mechanism comprises the following steps:

[0007] Step 1: 3D point cloud data preprocessing;

[0008] Step 2: Construct a 3D point cloud upsampling network structure based on the attention mechanism;

[0009] Step 3: Establish the loss function between the target point cloud and the output point cloud;

[0010] Step 4: Use public data sets for training and testing, and output the final results.

[0011] Step 1: Preprocessing of 3D point cloud data. The specific steps are as follows:

[0012] 1-1. Data preparation and downsampling. First, perform FPS downsampling on each 3D object in the original data set to obtain 2048 points to form a new data set 1. Then perform FPS sampling on data set 1 to obtain 256 points to form a new data set 2. Data set 2 containing 256 points is used as the input point cloud, and data set 1 containing 2048 points is used as the target point cloud. Then divide data set 1 and data set 2 into test data and training data at a ratio of 1:4 respectively.

[0013] 1-2. Perform data enhancement on the dataset, including rotation, scaling, jittering, random discarding, and adding Gaussian noise.

[0014] 1-3. Standardize the data set to make it satisfy the Gaussian distribution.

[0015] Step 2: Construct a 3D point cloud upsampling network structure based on the attention mechanism. The specific steps are as follows:

[0016] 2-1. Build the encoder.

[0017] The encoder includes four feature extraction modules, two local feature fusion modules, and a maximum pooling layer. Each feature extraction module consists of a convolutional (Conv) block and a dense graph block, where the convolution block consists of a Conv layer, a batch normalization layer (BatchNormalization, BN) and a linear rectifier function (ReLU), and the graph convolution block consists of a Graph-Conv layer, BN and ReLU. Each feature extraction module is connected by a skip connection. Each local feature fusion module consists of a k-nearest neighbor aggregation layer (knn-downsampling), two Conv layers, two BNs, two ReLUs and an attention module. The attention module uses the input feature vector (N*C, where N is the number of input three-dimensional points and C is the dimension of each point) as the query vector Q and the key value vector V, respectively, and obtains the attention score (N*N) through matrix multiplication and softmax function. Then, the input feature vector (N*C) is multiplied by the attention score (N*N) to obtain the optimized feature vector, which further integrates the local features. The optimized feature vector is then max-pooled, that is, the maximum value of the features among N points is taken to obtain the global feature (1*C). Finally, the global feature is copied N times and concatenated with the feature vector before max-pooling to obtain the encoded vector (2N*C), which will be further input into the feature expansion module.

[0018] 2-2. Construct feature expansion module.

[0019] The feature expansion module copies the feature (2N*C) output by the encoder by r times (i.e., 2rN*C), where r is the upsampling ratio. Then, a position vector is generated. Specifically, the position vector size is (2rN*1), and its value is generated by uniform distribution between -1 and 1. Finally, the copied vector (2rN*C) is concatenated with the position vector (2rN*1) to obtain the expanded feature vector (2rN*(C+1)).

[0020] 2-3. Build a decoder.

[0021] The decoder consists of three fully connected layers and a point cloud reconstruction layer. Each fully connected layer includes a Conv layer, a Leaky-ReLU function, and a BN layer. The decoder input and the point cloud reconstruction layer use a jump connection, and the point cloud reconstruction layer converts the decoded features back into three-dimensional coordinate points.

[0022] Step 3: Establish the target loss function between the target point cloud and the output point cloud, as follows:

[0023] 3-1. Since Euclidean distance is fast and flexible in calculation, Euclidean distance is used as the point cloud distance standard. It can be expressed in mathematical form as follows:

[0024]

[0025] Where X(1*d) and Y(1*d) are any two different points, where d is the dimension of each point; i ,y i They are the component vectors (1*1) of the points respectively.

[0026] 3-2. Calculate the distance between the output point cloud and the target point cloud.

[0027] The Chamfer Distance formula is used as the loss function for upsampling the 3D point cloud to minimize the distance between the output point cloud and the target point cloud, which is defined as follows:

[0028]

[0029] In the above formula, S1 and S2 represent the output point cloud and the target point cloud respectively. The first term Represents the sum of the minimum distances from any point x in S1 to S2. The second term It represents the sum of the minimum distances from any point y in S2 to S1. The smaller the distance, the smaller the difference between the output point cloud and the target point cloud.

[0030] Step 4: Train the model and test the data as follows:

[0031] 4-1. Prepare the original dataset (such as public datasets ModelNet10, ModelNet40, ShapeNet, Shrec16) and preprocess it as described in step 1.

[0032] 4-2. Input the training data into the constructed 3D point cloud upsampling network for training and use the test data for testing.

[0033] 4-3. In order to verify the efficiency of the method of the present invention, it is compared with the current excellent methods (such as PU-NET, MPU, etc.), and the CD (Chamfer Distance), EMD (Earth Mover's Distance) and the reliability and practicality of the analysis results are calculated.

[0034] The beneficial effects of the present invention are:

[0035] This method proposes a 3D point cloud upsampling network based on the attention mechanism, further strengthens the learning of local features and neighborhood relationships, and adds a residual structure to optimize the information transfer process, improves the accuracy of 3D point cloud upsampling, and reduces the loss between the target point cloud and the upsampled point cloud. The experimental results clearly confirm the efficiency and practicality of the proposed method. In short, the proposed method is more efficient and accurate for 3D point cloud upsampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the steps of the present invention;

[0037] Figure 2 It is a flow chart of the overall architecture of the present invention;

[0038] Figure 3 It is a flow chart of the feature extraction unit of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0040] 3D point cloud upsampling method based on attention mechanism, for specific steps, see Figure 1 As shown in the overall architecture flow chart Figure 2 As shown:

[0041] A 3D point cloud upsampling method based on attention mechanism, the steps are as follows:

[0042] Step 1: Preprocessing of 3D point cloud data. The specific steps are as follows:

[0043] 1-1. Data preparation and downsampling. First, perform the farthest distance downsampling (FPS) on each 3D object in the original data set (such as ModelNet40) to obtain 2048 points to form a new data set 1, and then perform FPS sampling in data set 1 to obtain 256 points to form a new data set 2. Data set 1 containing 256 points is used as the input point cloud, and the data set containing 2048 points is used as the target point cloud. Then divide data set 1 and data set 2 into a ratio of 1:4, with one as test data and the other as training data. The advantage of the FPS method over random sampling is that it can cover all points in space as much as possible. The specific implementation method is as follows:

[0044] Assume that there are p points in total, and the entire point set H = {h1,h2,...,h p}, the goal is to select q points so that the q points are the farthest apart from each other: ① Randomly select a point h i As the starting point, write the starting point set B = {h i};②Select the remaining n-1 points to calculate and h iThe distance between the points, select the farthest point h i Write starting point set B = {h i ,h j}; ③ Select the remaining n-2 points to calculate the distance to each point in point set B, and take the shortest distance as the distance from the point to the point set. In this way, n-2 distances to the point set are obtained, and the farthest point is selected and written into the starting point B = {h i ,h j ,h k}, and n-3 points are left. If q=3, the selection is complete; if q>3, repeat the above steps until q starting points are selected.

[0045] 1-2. Perform data enhancement on the dataset, including random rotation, scaling, jittering, random discarding, and adding Gaussian noise.

[0046] 1-3. Standardize the data set. Standardizing the data set is to scale the data so that it is mapped to a unit sphere with a radius of 1:

[0047]

[0048] Among them, S is the point cloud to be standardized, x i is any point in S.

[0049] Step 2: Construct a 3D point cloud upsampling network structure based on the attention mechanism. The specific steps are as follows:

[0050] 2-1. Build the encoder.

[0051] The encoder includes four feature extraction modules, two local feature fusion modules, and a maximum pooling layer. Each feature extraction module consists of a convolutional (Conv) block and a dense graph block, where the convolution block consists of a Conv layer, a batch normalization layer (BatchNormalization, BN) and a linear rectifier function (ReLU). The convolution block maps the input feature vector to a 24-dimensional space and outputs it to the graph convolution block. The graph convolution block consists of a Graph-Conv layer, BN and ReLU, and the graph convolution block maps the input feature vector to a 60-dimensional vector space. Each feature extraction module is connected by a skip connection to reduce the information loss caused by the network depth. After four feature extraction modules and skip connections, a 256*264 feature vector (256 points, each point has a dimension of 264) is obtained and input into the local feature fusion module. Each local feature fusion module consists of a k-nearest neighbor aggregation layer (knn-downsampling), two Conv layers, two BNs, two ReLUs, a maximum pooling layer, and an attention module. The k-nearest neighbor aggregation layer takes each input point (1*264) as the center, selects 32 points with the closest distance, and then performs feature fusion and maximum pooling through two Conv layers. The obtained features are input into the self-attention module and output a 256*1024 feature vector. The self-attention mechanism is similar to how humans always focus on key information when seeing a picture. It can more accurately discover the important parts of the feature vector and increase its weight, so it can be used as a module to optimize the network. Specifically, the attention module uses the input feature vector (N*C, where N is the number of input three-dimensional points and C is the dimension of each point) as the query vector Q and the key value vector V, respectively, and obtains the attention score (N*N) through matrix multiplication and softmax function. Then, the input feature vector (N*C) is multiplied by the attention score (N*N) to obtain the optimized feature vector, which further integrates the local features. Then the feature vector is max-pooled, that is, the maximum value of the features among N points is taken to obtain the global feature (1*C). Finally, the global feature is copied N times (N*C) and concatenated with the feature vector before max-pooling to obtain the encoded vector (2N*C), which will be further input into the feature expansion module.

[0052] 2-2. Construct feature expansion unit.

[0053] The feature expansion module copies the feature (2N*C) output by the encoder by r times (i.e., 2rN*C), where r is the upsampling ratio (e.g., 8). Then, a position vector is generated. Specifically, the position vector size is (2rN*1), and its value is generated by uniform distribution between -1 and 1. Finally, the copied vector (2rN*C) is concatenated with the position vector (2rN*1) to obtain the expanded feature vector (2rN*(C+1)).

[0054] 2-3. Build a decoder.

[0055] The decoder consists of three fully connected layers and a point cloud reconstruction layer. Each fully connected layer includes a Conv layer, a Leaky-ReLU function, and a BN layer. The output of each layer is (r*1024), (r*512), and (3*2048). The input of the decoder is connected to the point cloud reconstruction layer (explanation) using a jump connection. The point cloud reconstruction layer converts the decoded features back into three-dimensional coordinate points (completed by matrix transposition), and the output is 2048*3.

[0056] Step 3: Establish the target loss function between the target point cloud and the output point cloud, as follows:

[0057] 3-1. Since Euclidean distance is fast and flexible in calculation, Euclidean distance is used as the point cloud distance standard. It can be expressed in mathematical form as follows:

[0058]

[0059] Where X(1*d) and Y(1*d) are any two different points, where d is the dimension of each point; i ,y i They are the component vectors (1*1) of the points respectively.

[0060] 3-2. Calculate the distance between the output point cloud and the target point cloud.

[0061] The Chamfer Distance formula is used as the loss function for upsampling the 3D point cloud to minimize the distance between the output point cloud and the target point cloud, which is defined as follows:

[0062]

[0063] In the above formula, S1 and S2 represent the output point cloud and the target point cloud respectively. The first term Represents the sum of the minimum distances from any point x in S1 to S2. The second term It represents the sum of the minimum distances from any point y in S2 to S1. The smaller the distance, the smaller the difference between the output point cloud and the target point cloud.

[0064] Step 4: Use the public data set for training and testing, and output the final results, as follows:

[0065] 4-1. Prepare a dataset (such as public datasets ModelNet10, ModelNet40, ShapeNet, Shrec16), and then preprocess it as described in step 1.

[0066] 4-2. Input the training data into the 3D point cloud upsampling network for training and use the test data for testing. The loss function is Chamfer Distance, the optimizer is Adam, the initial learning rate is 0.0001, the number of training cycles is 300, the batch size is 32, and the hardware is NVIDIA RTXA5000.

[0067] 4-3. In order to verify the efficiency of the method of the present invention, it is compared with the current excellent methods (such as PU-NET, MPU and other methods) and some classic encoder models (such as PointNet, SONet and other methods). The evaluation indicators include CD (Chamfer Distance) and EMD (Earth Mover's Distance). The smaller the value, the better the model effect. At the same time, the SVM classification accuracy Cls and retrieval accuracy MAP of the model are tested. The larger the value, the better the model effect.

[0068] Experimental results:

[0069] 1. The results of this method and PointNet, TOPNet, Deconvolution, SONet, PSGNet, PDGN, FoldingNet, PUNet, AtlasNet, and MPU methods were tested when the number of input point clouds was 128, and different evaluation indicators (CD, EMD, Cls, MAP) were compared. See Table 1 for specific data results.

[0070] Table 1 Comparative experimental results of this method and other methods when the number of input point clouds is 128

[0071]

[0072] 2. The results of this method and PointNet, TOPNet, Deconvolution, SONet, PSGNet, PDGN, FoldingNet, PUNet, AtlasNet, and MPU methods were tested when the number of input point clouds is 256, and different evaluation indicators (CD, EMD, Cls, MAP) were compared. See Table 2 for specific data results.

[0073] Table 2 Comparative experimental results of this method and other methods when the number of input point clouds is 256

[0074]

Claims

1. A 3D point cloud upsampling method based on attention mechanism, characterized in that: The following steps are involved: Step 1: 3D point cloud data preprocessing; Step 2: Construct a 3D point cloud upsampling network structure based on the attention mechanism; Step 3: Establish the loss function between the target point cloud and the output point cloud; Step 4: Use public data sets for training and testing, and output the final results; Step 2 The specific steps are as follows: 2-1. Build the encoder; The encoder includes four feature extraction modules, two local feature fusion modules, and a maximum pooling layer; each feature extraction module consists of a convolution block and a graph convolution block, where the convolution block consists of a Conv layer, a batch normalization layer, and a linear rectification function, and the graph convolution block consists of a Graph-Conv layer, BN, and ReLU. Each feature extraction module is connected by a jump connection; each local feature fusion module consists of a k-nearest neighbor aggregation layer, two Conv layers, two BNs, two ReLUs, and an attention module; the attention module takes the input feature vector N*C, where N is the number of input three-dimensional points, C is the dimension of each point, and they are used as query vector Q and key value vector V respectively. After matrix multiplication and softmax function, the attention score N*N is obtained. Then, the input feature vector N*C is multiplied by the attention score N*N to obtain the optimized feature vector, which further integrates the local features. Then, the optimized feature vector is max-pooled, that is, the maximum value of the features among N points is taken to obtain the global feature 1*C. Finally, the global feature is copied N times and concatenated with the feature vector before max-pooling to obtain the encoded vector 2N*C, which will be further input into the feature expansion module. 2-2. Construct feature expansion module; The feature expansion module copies the feature 2N*C output by the encoder by r times, that is, 2rN*C, where r is the upsampling ratio; then generates a position vector. Specifically, the position vector size is 2rN*1, and its value is generated by uniform distribution between -1 and 1; finally, the copied vector 2rN*C is concatenated with the position vector 2rN*1 to obtain the expanded feature vector 2rN*(C+1); 2-3. Build a decoder; The decoder consists of three fully connected layers and a point cloud reconstruction layer; each fully connected layer includes a Conv layer, a Leaky-ReLU function, and a BN layer; a jump connection is used between the decoder input and the point cloud reconstruction layer, and the point cloud reconstruction layer reconverts the decoded features into three-dimensional coordinate points; Step 3 is as follows: 3-1. Since the Euclidean distance is fast and flexible in calculation, the Euclidean distance is used as the point cloud distance standard; it is expressed in mathematical form as follows: Where X and Y are any two different points, and d is the dimension of each point; i ,y i are the component vectors of a point respectively; 3-2. Calculate the distance between the output point cloud and the target point cloud; The chamfer distance formula is used as the loss function for upsampling the 3D point cloud to minimize the distance between the output point cloud and the target point cloud, which is defined as follows: In the above formula, S1 and S2 represent the output point cloud and the target point cloud respectively. The first term Represents the sum of the minimum distances from any point x in S1 to S2. The second term It represents the sum of the minimum distances from any point y in S2 to S1; the smaller the distance, the smaller the difference between the output point cloud and the target point cloud.

2. According to the three-dimensional point cloud upsampling method based on the attention mechanism of claim 1, it is characterized in that: Step 1 The specific steps are as follows: 1-1. Data preparation and downsampling: First, perform FPS downsampling on each 3D object in the original data set at the farthest distance to obtain 2048 points to form a new data set 1, and then perform FPS sampling on data set 1 to obtain 256 points to form a new data set 2; data set 2 containing 256 points is used as the input point cloud, and data set 1 containing 2048 points is used as the target point cloud; then data set 1 and data set 2 are divided into test data and training data at a ratio of 1:4 respectively; 1-2. Perform data enhancement on the dataset, including rotation, scaling, jitter, random discarding, and adding Gaussian noise; 1-3. Standardize the data set to make it satisfy the Gaussian distribution.

3. The three-dimensional point cloud upsampling method based on the attention mechanism according to claim 2, characterized in that: Step 4 is as follows: 4-1. Prepare the original data set and perform preprocessing according to step 1; 4-2. Input the training data into the constructed 3D point cloud upsampling network for training and use the test data for testing; 4-3. Calculate CD, EMD and analyze the reliability and practicality of the results.

Images