A tactile-assisted point cloud super-resolution method

Abstract

The present invention belongs to the field of three-dimensional point cloud super-resolution, and specifically relates to a tactile-assisted three-dimensional point cloud super-resolution method based on machine learning. It overcomes the problem of poor edge effect of reconstructed high-resolution point cloud due to too little input of low-resolution point cloud information during high-magnification point cloud super-resolution, and uses local tactile information to improve the point cloud super-resolution effect. It mainly includes the following steps: Step 1, construct a point cloud super-resolution dataset containing local tactile information; Step 2, use the constructed point cloud feature extraction module to extract features and fuse the low-resolution point cloud and local tactile point cloud input to the neural network; Step 3, use the dataset to train the neural network in a supervised learning manner; Step 4, input the low-resolution point cloud and the local tactile point cloud into the trained network to generate a high-resolution point cloud.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud super-resolution, and in particular relates to a tactile-assisted point cloud super-resolution method based on machine learning. Background Art

[0002] Point cloud is a collection of points in space. It is the result of digital sampling of the real world and contains rich information. In recent years, with the gradual maturity of technologies such as face recognition, autonomous driving, and 3D modeling, there is an increasing demand for high-quality point cloud data processing. However, limited by the current level of sensor chip technology, the directly collected 3D point cloud data usually has a low resolution, far less than visible light images; at the same time, the material and color of the object itself will affect its reflectivity, thereby affecting the imaging quality of the laser scanner when facing objects of different colors and materials. This will result in low quality and density of the acquired point cloud, which is difficult to meet the use requirements. In order to improve the resolution of point cloud data, many scholars at home and abroad have conducted research on point cloud data super-resolution algorithms. In recent years, with the development of high-performance computing devices and the continuous improvement of machine learning technology, super-resolution methods based on machine learning have attracted more and more attention from scholars at home and abroad. PU-Net first applied machine learning to the work of point cloud data super-resolution. The core idea of ​​PU-Net is to learn the multi-level features of each point from local to global, then expand the point set in the feature space, and finally map the expanded point set back to three dimensions. Many subsequent methods are developed on this basis. Inspired by the excellent performance of Transformer in natural language processing, the point cloud super-resolution network PU-Transformer based on the encoder-decoder framework was proposed. The core of this method is also to learn the representative features of a given point to estimate the distribution of new points. It uses the encoder-decoder framework, the attention mechanism, and the position encoding method to process the features of the three-dimensional point cloud, and finally reconstructs the high-resolution point cloud through randomization and fully connected layers. Due to the discreteness of point clouds and the unstructured local area of ​​point cloud prediction, the above method is difficult to maintain a good distribution structure of points in the local area, and thus cannot complete the point cloud super-resolution task well at locations such as edges.

[0003] Touch is another way to perceive the 3D shape of an object. For robots, most tactile sensors measure the geometry of the contact surface. The robot can reconstruct the shape of an object by multiple touches, combining the position and posture of the sensor at each touch, without being affected by the ambiguity caused by the color or material of the object's surface. However, tactile information is limited by the size and scale of the sensor: since each touch can only obtain information about a local area, it takes multiple touches and a long time to reconstruct the complete shape of the object, so a single point cloud reconstruction based only on touch is difficult to be applied in practice.

[0004] Therefore, in the field of 3D point cloud super-resolution, it is currently necessary to explore a point cloud super-resolution method that can not only utilize the learning ability of neural networks, but also reasonably utilize the local touch information of objects, so as to improve the effect of point cloud super-resolution. Summary of the invention

[0005] Existing 3D point cloud super-resolution methods have certain limitations. For example, the point cloud super-resolution method based on neural network uses low-resolution point cloud as network input. When performing high-magnification point cloud super-resolution, it is often difficult to capture local geometric detail structures due to too few input points, resulting in poor point cloud super-resolution effect for complex objects; and although object reconstruction based on tactile information alone can obtain dense point cloud information, it is limited by the size and scale of the sensor at each touch: since each touch can only obtain information of a local area, it is often inefficient to obtain a high-resolution point cloud in this way. In the present invention, a set of 3D point cloud super-resolution methods assisted by local tactile point clouds is proposed. This method overcomes the problem of poor edge effect of the reconstructed high-resolution point cloud due to too little input of low-resolution point cloud information when performing super-resolution on the point cloud, and improves the point cloud super-resolution effect by using local tactile information through feature fusion.

[0006] The present invention provides a method for tactile-assisted point cloud super-resolution based on machine learning, comprising the following steps:

[0007] Step 1, constructing a point cloud super-resolution dataset, wherein the dataset includes a local tactile point cloud, a low-resolution point cloud, and a high-resolution point cloud;

[0008] Step 2: construct a point cloud super-resolution network;

[0009] Step 3: Use the dataset to train the point cloud super-resolution network in a supervised learning manner. The local tactile point cloud and low-resolution point cloud in the dataset are used as the input of the neural network, and the high-resolution point cloud information in the dataset is used as the output of the neural network.

[0010] The forward and back propagation of the neural network are used to optimize the overall network parameters, and the point cloud super-resolution network training is constrained by the loss function, so that the neural network fits the training set data and obtains the trained point cloud super-resolution network;

[0011] Step 4: Input the collected low-resolution point cloud and local tactile point cloud into the trained point cloud super-resolution network to generate a high-resolution point cloud.

[0012] To verify the effect of tactile-assisted point cloud super-resolution, several experiments were conducted, including the effect of adding touch point cloud on point cloud super-resolution, the effect of adding different times of touch on point cloud super-resolution, and the effect of different feature fusion methods on point cloud super-resolution. The object model used in the experimental stage was selected from the test data set constructed in step 1. The low-resolution point cloud was input into the point cloud super-resolution network to achieve super-resolution of the low-resolution point cloud.

[0013] Furthermore, the method for constructing the data set in step 1 is as follows:

[0014] Based on the 3D reconstruction dataset, the real value point cloud in the 3D reconstruction dataset is used as the high-resolution point cloud;

[0015] Extracting part of the data in the 3D reconstruction data set as a local tactile point cloud;

[0016] And generate low-resolution point cloud data by sampling the farthest point of the high-resolution point cloud;

[0017] Finally, the local tactile point cloud is aligned with the location of the object and added to the dataset, and finally the dataset is converted into .h5 format to complete the production of the dataset; the number of point clouds of the true value GT (Groud Truth, GT) point cloud is M×R points, the sampled low-resolution point cloud is M points, and the local tactile point cloud is uniformly sampled into N points.

[0018] Furthermore, the point cloud super-resolution network includes two point cloud feature extraction modules and a Transformer network;

[0019] The local tactile point cloud is input into a point cloud feature extraction module for feature extraction to obtain local tactile point cloud features;

[0020] The low-resolution point cloud is sent to another point cloud feature extraction module for feature extraction to obtain low-resolution point cloud features;

[0021] The extracted local tactile point cloud features and low-resolution point cloud features are fused to obtain the overall features, which are then input into the Transformer network. At the same time, the low-resolution point cloud is also input into the Transformer network, and the output features of the Transformer network are reconstructed into a high-resolution point cloud through randomization and fully connected layers.

[0022] The Transformer network includes n attention encoders connected in series, and the low-resolution point cloud is input into each attention encoder in the Transformer network.

[0023] Further, the point cloud feature extraction module includes a first double convolution layer, a first maximum pooling layer, a second double convolution layer, a second maximum pooling layer, a maximum pooling layer, a third double convolution layer, a first deconvolution layer, and a second deconvolution layer connected in sequence;

[0024] The point cloud generates intermediate feature I after passing through the first double convolution layer, and then generates intermediate feature II after passing through the first maximum pooling layer and the second double convolution layer;

[0025] After the second maximum pooling layer and the third double convolution layer, the intermediate feature III is generated;

[0026] The intermediate feature III is input to the first deconvolution layer and concatenated with the intermediate feature II, and output as the intermediate feature IV;

[0027] The intermediate feature IV is then concatenated with the intermediate feature I after passing through the second deconvolution layer to generate the final point cloud feature.

[0028] Furthermore, the double convolution layer in the point cloud feature extraction module is used as the basic point cloud feature extraction structure to increase the number of channels of feature information, and the maximum pooling layer is used to change the size of the feature.

[0029] Furthermore, the first double convolution layer, the second double convolution layer and the third double convolution layer in the point cloud feature extraction module have the same structure. The double convolution layer structure is the basic point cloud feature extraction structure;

[0030] The double convolutional layer includes the first convolutional layer, the first batch of normalization layers, the second convolutional layer and the second batch of normalization layers connected in sequence, where each convolution layer uses ReLU as the activation function.

[0031] Furthermore, in step 3, when the point cloud super-resolution network is trained using the data set, the loss function used by the network is the Chamfer Distance (CD), which is expressed as:

[0032]

[0033] Among them, S1 and S2 are two groups of point clouds that need to be compared. The chamfer distance can effectively measure the difference between the two groups. The larger the value, the greater the difference between the two groups of point clouds.

[0034] Furthermore, when the extracted local tactile point cloud features and the low-resolution point cloud features are fused, the ratio of the local tactile point cloud features to the low-resolution point cloud features is 1:4.

[0035] Furthermore, in step 4, the low-resolution point cloud and the local tactile point cloud are input into the trained network to generate a high-resolution point cloud (the number of points is M×N). The object model used in the experimental stage comes from the test set part of the dataset we created in step 1.

[0036] This method uses the Transformer network and feature fusion module to achieve super-resolution of low-resolution point clouds, and uses the test results in the original Transformer network as a benchmark to conduct comparative experiments with the method of the present invention. In order to verify the experimental effect of tactile-assisted point cloud super-resolution, we first conducted a point cloud super-resolution experiment when adding touch, and also compared the impact of different feature extraction methods on the experimental results when adding local tactile point clouds on the data set. In the experiment, all low-resolution point clouds can be successfully reconstructed, and most objects can be reconstructed better using the correct feature fusion method when adding tactile information. In particular, for more complex objects, before adding local tactile point clouds, the edge details cannot be well reconstructed, and better details can be reconstructed after using tactile assistance.

[0037] Furthermore, in step 4, point cloud super-resolution experiments were conducted using different feature fusion methods. The experiment used the test set of the data set for testing, and used three different methods in turn: direct splicing of point clouds, direct splicing of features, and feature extraction and fusion using the point cloud feature extraction module constructed in the present invention, and compared with the test results of the Transformer network without adding touch. Because the local tactile point cloud is too dense, directly splicing the local tactile point cloud and the low-resolution point cloud cannot achieve good results; directly splicing features can achieve better experimental results, but the improvement is limited compared to not adding touch; using the point cloud feature extraction module in the present invention to extract the features of the local tactile point cloud and the input point cloud respectively and then sending them to the super-resolution network can reconstruct better details and achieve better results on the test set; this is because the feature extraction can correctly utilize the local features of the local tactile point cloud, so that the details of the object can be better reconstructed.

[0038] Furthermore, in step 4, a point cloud super-resolution experiment was conducted when different touch times were added. The experiment was tested using the test set of the dataset, and the point clouds with 1 and 2 touches were added in turn, and compared with the test results of the Transformer network without adding touches. After correct feature fusion, the super-resolution effect of objects with more local tactile point clouds is better.

[0039] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses the data set to pre-train a point cloud super-resolution network model in a supervised learning manner, and after the training is completed, only low-resolution point cloud and local tactile point cloud information need to be input to complete point cloud super-resolution, and the reconstruction effect is better than the case where local tactile point cloud information is not added. The method is highly feasible and universal, and the effect is more significantly improved when performing point cloud super-resolution on complex objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the method of the present invention.

[0041] Figure 2 This is the overall framework diagram of the point cloud super-resolution network.

[0042] Figure 3 This is the framework diagram of the point cloud feature extraction module.

[0043] Figure 4 It is a double convolutional layer structure diagram.

[0044] Figure 5 This is the point cloud super-resolution effect after adding touch. DETAILED DESCRIPTION

[0045] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0046] The method of tactile-assisted point cloud super-resolution based on machine learning of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0047] Step 1, construct a point cloud super-resolution dataset containing tactile information;

[0048] The dataset is based on a 3D reconstruction dataset containing tactile information. The GT point cloud in the 3D reconstruction dataset is used as a high-resolution point cloud, and low-resolution point cloud data is generated by sampling the farthest point of the high-resolution point cloud. Finally, the local tactile point cloud is aligned with the location of the object and added to the dataset to complete the production of the dataset.

[0049] In this embodiment, the number of points of the high-resolution point cloud (GT) in the data set is 8192, the number of points of the sampled low-resolution point cloud is 512, and the local tactile point cloud is uniformly sampled into 1000 points (the number of points can be adjusted arbitrarily according to actual needs).

[0050] Step 2: construct a point cloud super-resolution network;

[0051] The point cloud super-resolution network includes two point cloud feature extraction modules and a Transformer network;

[0052] The point cloud feature extraction module is used to convert the low-resolution point cloud and the local tactile point cloud into low-resolution point cloud features and local tactile point cloud features and fuse them, which specifically includes the following steps:

[0053] Step 2.1, construct a double convolutional layer structure, such as Figure 3As shown in the figure, each double convolutional layer (Conv[D]) contains two convolutional layers (Conv) and two batch normalization layers (BN); each convolution layer uses ReLU as the activation function.

[0054] Step 2.2, build a point cloud feature extraction module, such as Figure 4 As shown; the point cloud feature extraction module is based on the basic structure constructed in step 2.1. The input point cloud passes through a double convolution layer (Conv[D]I) to generate an intermediate feature I, wherein the double convolution layer (Conv[D]) is an expanded convolution, which can increase the number of feature information channels; then the feature is input into the first layer of the maximum pooling layer, which will only change the size of the feature but not the number of feature information channels. The feature will be output to the double convolution layer (Conv[D]II) to generate an intermediate feature II; the intermediate feature II will pass through the second layer of the maximum pooling layer and the double convolution layer (Conv[D]III) to generate an intermediate feature III; the intermediate feature III will be input into the deconvolution layer and concat with the intermediate feature II, and its output is the intermediate feature IV; finally, the intermediate feature IV will be deconvolved and concat with the intermediate feature I to generate the final point cloud feature.

[0055] In step 2.3, the input low-resolution point cloud and local tactile point cloud are respectively sent to the point cloud feature extraction module to extract the low-resolution point cloud features and the local tactile point cloud features. Finally, the low-resolution point cloud features and the local tactile point cloud features are fused to complete the feature extraction.

[0056] Step 3: Use the dataset to train the Transformer network in a supervised learning manner, such as Figure 2 As shown; specifically includes the following steps:

[0057] Step 3.1, the tactile point cloud super-resolution dataset obtained in step 1 is made into a dataset in .h5 format for network reading.

[0058] Step 3.2: Input the features generated in step 2 into the Transformer network backbone, and the low-resolution point cloud is also input into each attention encoder. In this embodiment, the number of attention encoders n is 5, which uses a cascade method to process the features and input point clouds; the loss function used is the chamfer distance, which is expressed as:

[0059]

[0060] Among them, S1 and S2 are two groups of point clouds that need to be compared, which are the generated high-resolution point cloud and GT point cloud respectively; the chamfer distance can effectively measure the difference between the two groups. The larger the value, the greater the difference between the two groups of point clouds.

[0061] Step 4: supervised training is performed using the high-resolution point cloud pair corresponding to the low-resolution point cloud as a label. After the training is completed, a training model is obtained, and then a high-resolution point cloud is generated through the model.

[0062] To verify the effect of tactile-assisted point cloud super-resolution, several experiments were conducted, including the effect of adding touch point cloud on point cloud super-resolution, the effect of adding different times of touch on point cloud super-resolution, and the effect of different feature fusion methods on point cloud super-resolution. The object model used in the experimental stage was selected from the test data set constructed in step 1. The low-resolution point cloud was input into the overall network to achieve super-resolution of the low-resolution point cloud.

[0063] When targeting specific objects, the experimental results of the effect of adding different touch times on point cloud super-resolution are shown in Table 1. The evaluation criterion of the data in Table 1 is the chamfer distance between the reconstructed high-resolution point cloud and the real point cloud. The smaller the value, the better the super-resolution reconstruction effect.

[0064] Table 1 Chamfer distance (CD) between high-resolution point cloud obtained by super-resolution of typical samples and the real point cloud

[0065]

[0066] Table 1 shows the chamfer distance between the reconstructed high-resolution point cloud and the real point cloud, taking nuts, tires, and workpieces as examples. It can be found that adding touch can effectively reduce the chamfer distance between the reconstruction result and the real point cloud, and as the number of added touches increases, the chamfer distance decreases. This shows that adding tactile information can effectively assist the point cloud super-resolution task. Figure 5 The super-resolution effect of nuts, tires, and workpieces is shown. The data of each type of object in the figure is represented as follows: Figure 5 (a) Real point cloud; (b) Low-resolution point cloud with secondary touch; (c) Super-resolution result without touch; (d) Super-resolution result with secondary touch. As can be seen from the figure, all objects can be reconstructed, and all objects can be reconstructed better after adding tactile information. For more complex objects, such as nuts and tires, the method can achieve better reconstruction of the object edges.

[0067] Table 2 Chamfer distance (CD) between high-resolution point cloud obtained by super-resolution of different tactile-assisted methods and the real point cloud

[0068]

[0069] When testing with the test set, the comparison of experimental results using different fusion methods and different touch times is shown in Table 2. The evaluation criteria of the data in Table 2 is the chamfer distance between the reconstructed high-resolution point cloud and the real point cloud. The smaller the value, the better the super-resolution effect. Table 2 shows the chamfer distance of point cloud super-resolution using three different fusion methods. It can be found that directly splicing point clouds cannot improve the performance of point cloud super-resolution. This is because the local tactile point cloud is too dense, and directly splicing the local tactile point cloud and the low-resolution point cloud cannot achieve good results; while directly splicing features can achieve better experimental results, but the improvement is limited compared to not adding tactile. Using the point cloud feature extraction module to extract the features of the local tactile point cloud and the input point cloud respectively and then send them to the super-resolution network can reconstruct better details and achieve better results on the test set; this is because the feature extraction can correctly use the local features of the local tactile point cloud, so that the details of the object can be better reconstructed. Table 2 also shows the performance of the method on the test set when using different touch times. It can be found that when the correct feature extraction method is used, the chamfer distance decreases as the number of added touches increases. This shows that adding tactile information can effectively assist the point cloud super-resolution task.

Claims

1. A tactile-assisted point cloud super-resolution method based on machine learning, characterized in that: The steps include: Step 1, constructing a point cloud super-resolution dataset, wherein the dataset includes a local tactile point cloud, a low-resolution point cloud, and a high-resolution point cloud; Step 2: construct a point cloud super-resolution network; The point cloud super-resolution network includes two point cloud feature extraction modules and a Transformer network; The local tactile point cloud is input into a point cloud feature extraction module for feature extraction to obtain local tactile point cloud features; The low-resolution point cloud is sent to another point cloud feature extraction module for feature extraction to obtain low-resolution point cloud features; The extracted local tactile point cloud features and low-resolution point cloud features are fused to obtain the overall features, which are then input into the Transformer network. The low-resolution point cloud is also input into the Transformer network, and the output features of the Transformer network are reconstructed into a high-resolution point cloud through randomization and a fully connected layer. The point cloud feature extraction module includes a first double convolution layer, a first maximum pooling layer, a second double convolution layer, a second maximum pooling layer, a maximum pooling layer, a third double convolution layer, a first deconvolution layer, and a second deconvolution layer connected in sequence; The point cloud generates intermediate feature I after passing through the first double convolution layer, and then generates intermediate feature II after passing through the first maximum pooling layer and the second double convolution layer; After the second maximum pooling layer and the third double convolution layer, the intermediate feature III is generated; The intermediate feature III is input to the first deconvolution layer and concatenated with the intermediate feature II, and output as the intermediate feature IV; The intermediate feature IV is then concatenated with the intermediate feature I after passing through the second deconvolution layer to generate the final point cloud feature. Step 3, using the dataset to train the point cloud super-resolution network in a supervised learning manner, the local tactile point cloud and low-resolution point cloud in the dataset are used as the input of the neural network, and the high-resolution point cloud information in the dataset is used as the output of the neural network; Step 4: Input the collected low-resolution point cloud and local tactile point cloud into the trained network to generate a high-resolution point cloud.

2. The method for tactile-assisted point cloud super-resolution based on machine learning according to claim 1, characterized in that: The method for constructing the dataset in step 1 is as follows: Based on the 3D reconstruction dataset, the real value point cloud in the 3D reconstruction dataset is used as the high-resolution point cloud; Extracting part of the tactile data in the three-dimensional reconstruction data set as a local tactile point cloud; Generate a low-resolution point cloud by sampling the farthest point of the high-resolution point cloud; Finally, the local tactile point cloud is aligned to the location of the object and added to the dataset.

3. The method for tactile-assisted point cloud super-resolution based on machine learning according to claim 1, characterized in that: The double convolution layer is used as the basic point cloud feature extraction structure to increase the number of channels of feature information, and the maximum pooling layer is used to change the size of the feature.

4. The method for tactile-assisted point cloud super-resolution based on machine learning according to claim 1, characterized in that: The first double convolutional layer, the second double convolutional layer and the third double convolutional layer have the same structure; The double convolutional layer includes the first convolutional layer, the first batch of normalization layers, the second convolutional layer and the second batch of normalization layers connected in sequence, where each convolution layer uses ReLU as the activation function.

5. The method for tactile-assisted point cloud super-resolution based on machine learning according to claim 1, characterized in that: When training the point cloud super-resolution network, the loss function used is the chamfer distance, expressed as: Among them, S1 and S2 are two sets of point clouds that need to be compared.

6. The method for tactile-assisted point cloud super-resolution based on machine learning according to claim 1, characterized in that: When the extracted local tactile point cloud features and low-resolution point cloud features are fused, the ratio of the local tactile point cloud features to the low-resolution point cloud features is 1:

4.

7. The method for tactile-assisted point cloud super-resolution based on machine learning according to claim 1, characterized in that: The Transformer network includes n attention encoders connected in series.

Images