A 3D point cloud super-resolution method based on multi-modal iterative fusion

Abstract

The present invention belongs to the field of 3D point cloud super-resolution, and discloses a 3D point cloud super-resolution method based on multimodal iterative fusion, which mainly includes the following steps: using a Unet feature extraction module to extract features from a low-resolution point cloud and a K-times tactile point cloud respectively, iteratively fusion the low-resolution point cloud features and the K-times tactile features, and sending the features after K-times iterative fusion together with the original low-resolution point cloud into a Transformer encoder module, and obtaining a high-resolution point cloud after feature rearrangement and coordinate reconstruction; the iterative fusion includes that the input of the first iterative fusion is the low-resolution point cloud features and the first tactile features, and the input of the k+1th iteration is the features of the k-times iterative fusion plus the k-th tactile features. The present invention effectively improves the auxiliary role of multiple tactile information on the low-resolution point cloud super-resolution task, improves the quality of high-resolution point clouds, and observes the effects of different numbers of iterations in the experiment to obtain the optimal number of iterations K.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud super-resolution, and specifically relates to a three-dimensional point cloud super-resolution method based on multi-modal iterative fusion. Background Art

[0002] With the rapid development of 3D scanners and 3D point cloud acquisition technology, 3D point clouds are increasingly used in computer vision, robot guidance, industrial design, and other fields. As long as the point cloud is dense enough, a sufficiently accurate model can be constructed to meet the needs of various tasks. However, due to limitations such as sensor resolution, scanning time, and scanning conditions, the collected point clouds are often relatively sparse. Existing point cloud upsampling methods only process a single low-resolution point cloud. When upsampling a point cloud that is too sparse is performed at a large rate, the results are poor, and no information from other modalities is used for assistance.

[0003] Traditional point cloud super-resolution algorithms require strong prior shape information. The classic moving least squares algorithm has strong prior assumptions about the surface shape of the point cloud. In addition, due to the complexity and disorder of the point cloud, the classic algorithm performs poorly in domain migration and robustness. With the successful application of deep learning in the field of two-dimensional image super-resolution and the introduction of PointNet, the first deep learning network to process point cloud data, researchers began to focus on building deep learning networks to complete point cloud super-resolution tasks. PU-net is the first point cloud upsampling network based on deep learning, and has become the pioneering work of deep learning for point cloud super-resolution. Many subsequent methods have been developed based on this method. Inspired by the generative adversarial network GAN network, PU-GAN is the first to use a generative adversarial network to process point cloud data super-resolution. The generator upsamples the input point cloud and outputs it, and the discriminator identifies the output and outputs the confidence. PU-GCN is the first to introduce an upsampling module using the GCN network. GCN enables the upsampler to encode the spatial information of the point neighborhood and learn new points from the feature space, rather than simply copying the original points or copying points after different transformations. PU-Transformer first introduced the Transformer module in the point cloud super-resolution network, using the position fusion module to capture the local context information of the point cloud data, which can provide more position-related information of low-resolution points.

[0004] In addition to traditional 3D scanners such as LiDAR that can obtain 3D models, touch is another way to perceive 3D shapes. Most tactile sensors can measure force distribution or geometry on a small contact area. Combined with the position and posture of the sensor in each touch, it can assist in reconstructing the shape of the object. However, tactile perception is limited by the size and scale of the sensor. Each touch only obtains information in a local area, so it may take multiple touches and a long time to reconstruct the complete shape of the object. With the development of tactile sensors, common tactile sensors such as Gelsight and DIGIT can obtain the local geometry of the contact surface by touch. These high-resolution local geometric information are often used in 3D reconstruction tasks due to their accuracy. A new paradigm for reconstructing 3D models by combining visual information and tactile information has gradually attracted attention. Through visual information and tactile information, as well as prior knowledge of common object shapes learned from large-scale data sets, the shape of 3D objects can be accurately predicted. However, the fusion of tactile information with other modal information has not been explored to a large extent.

[0005] Therefore, in the field of 3D point cloud super-resolution, it is currently necessary to explore a deep learning network that integrates multimodal information and uses precise local information to improve the point cloud super-resolution effect. Summary of the invention

[0006] The technical problem to be solved by the present invention is to integrate tactile information with other modal information to improve the point cloud effect.

[0007] The present invention provides a three-dimensional point cloud super-resolution method based on multi-modal iterative fusion, comprising the following steps:

[0008] Step 1: construct the point cloud super-resolution dataset 3DVT;

[0009] The point cloud super-resolution dataset 3DVT includes high-resolution point clouds, low-resolution point clouds, and multiple tactile point clouds;

[0010] Step 2: Build a super-resolution network;

[0011] The super-resolution network is used to fuse the input low-resolution point cloud and multiple tactile point clouds to output a high-resolution point cloud;

[0012] Step 3, after pre-processing the low-resolution point cloud and K tactile point clouds in the point cloud super-resolution dataset 3DVT, input them into the super-resolution network to train the super-resolution network, where K is greater than or equal to 1; the K tactile point clouds are K tactile point clouds selected from the multiple tactile point clouds in the point cloud super-resolution dataset 3DVT, and are used to train the super-resolution network.

[0013] Step 4: Send the K tactile point clouds to be fused and the low-resolution point cloud into the trained super-resolution network to obtain a high-resolution point cloud.

[0014] Furthermore, the super-resolution network includes a feature extraction module, a feature fusion module, a feature expansion module and a coordinate reconstruction module; the extraction module includes a tactile point cloud feature extraction module and a low-resolution point cloud feature extraction module;

[0015] The low-resolution point cloud and K-times tactile point clouds are respectively input into the low-resolution point cloud feature extraction module and the tactile point cloud feature extraction module, and feature extraction is performed on the feature point clouds to obtain low-resolution point cloud features and tactile point cloud features; the low-resolution point cloud features and the tactile point cloud features are input into the feature fusion module for feature fusion to obtain fused features, which are then input into the feature expansion module, and the expanded features are input into the coordinate reconstruction module for reconstruction to obtain a high-resolution point cloud.

[0016] Furthermore, the feature extraction module adopts an encoder-decoder structure, and a jump connection is used between the encoder and the decoder; the tactile point cloud feature extraction module and the low-score point cloud feature extraction module have the same structure but different numbers of channels.

[0017] Further, the K tactile point clouds are input into the tactile point cloud feature extraction module to extract features from the feature point clouds. Specifically, each tactile point cloud is input into the tactile point cloud feature extraction module to obtain K tactile point cloud features, which are respectively recorded as k-th tactile point cloud features, wherein 1≤k≤K, and the steps are as follows:

[0018] The k-th tactile point cloud is subjected to two layers of convolution operations to obtain the first intermediate feature of the tactile point cloud, the maximum pooling layer downsamples the first intermediate feature of the tactile point cloud, and then undergoes two layers of convolution operations to obtain the second intermediate feature of the tactile point cloud, the maximum pooling layer downsamples the second intermediate feature of the tactile point cloud, and then undergoes two layers of convolution operations and a deconvolution layer operation, and then is spliced ​​with the second intermediate feature of the tactile point cloud, and then sent to two layers of cascaded convolution layers, and then after passing through the deconvolution layer, it is spliced ​​with the first intermediate feature of the tactile point cloud and sent to two layers of cascaded convolution layers, and finally the k-th tactile point cloud feature is obtained;.

[0019] Furthermore, the low-score point cloud features and the tactile point cloud features are input into a feature fusion module for feature fusion to obtain fused features. The specific steps are as follows:

[0020] When K = 1;

[0021] After the low-resolution point cloud features and the first tactile point cloud features are spliced, they are sent to the convolution layer and the maximum pooling layer in sequence, and the output features of the maximum pooling layer are spliced ​​with the low-resolution point cloud features again and a deconvolution operation is performed, and finally the first fused point cloud features are obtained;

[0022] When K>1, the following steps are also included:

[0023] The k-th fused features and the k-1th tactile point cloud features are taken as input, and after feature splicing, they pass through the convolution layer and the maximum pooling layer in sequence, and then spliced ​​with the original low-resolution point cloud features, and then pass through the deconvolution layer to obtain the k-th fused features. This cycle is repeated until the K-th fused features are obtained. The K-th fused features are the fused features output by the feature extraction module.

[0024] Furthermore, the feature expansion module is composed of a multi-layer cascaded Transformer encoder, which is used to extract high-dimensional features from the point cloud features after the K-th fusion, and input them as expanded features into the coordinate reconstruction module; in the coordinate reconstruction module, the expanded features are first rearranged to obtain rearranged features, and then the rearranged features are mapped back to the three-dimensional coordinate space through a multi-layer perception mechanism to obtain a high-resolution point cloud.

[0025] Furthermore, in the process of training the super-resolution network, the chamfer distance CD is used as the loss function to minimize the distance between the low-resolution point cloud and the high-resolution point cloud, as shown in the following formula:

[0026]

[0027] Where P represents the high-resolution point cloud predicted by the super-resolution network, Q represents the real high-resolution point cloud, p∈P represents the point in the predicted point cloud, q∈Q represents the point in the high-resolution point cloud in the dataset 3DVT, and the first term represents the sum of the minimum distances from any point p in P to Q, and the second term represents the sum of the minimum distances from any point q in Q to P.

[0028] Furthermore, in step 3, the low-resolution point cloud and the K-times tactile point cloud in the point cloud super-resolution dataset 3DVT are preprocessed, specifically, the tactile point cloud and the low-resolution point cloud are normalized, and data enhancement processing is performed.

[0029] Furthermore, the data enhancement processing includes rotating, scaling and randomly perturbing the normalized tactile point cloud and the low-score point cloud.

[0030] Furthermore, K is the optimal number of touches. Through iteration, the super-resolution effect of the point cloud is observed in the experiment when the number of touches takes different values, so as to obtain the optimal value K in the current data set.

[0031] Beneficial effects: Existing 3D point cloud super-resolution methods have certain limitations. When a too sparse point cloud is upsampled at a large rate, problems such as holes, abnormal points, and uneven distribution are prone to occur. This is because when a low-resolution point cloud is too sparse, it carries too little information. Tactile point clouds have the characteristics of high density and precision, but they have a small coverage area and can only obtain local area information. A large number of touches are required to obtain all the information about the object. The present invention makes full use of the advantages of tactile point clouds to assist in the super-resolution task of low-resolution point clouds, and fuses the low-resolution point cloud features with the tactile point cloud features, so that the point cloud features contain more local information, improving the quality of the output high-resolution point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 It is the structural diagram of feature extraction module;

[0034] Figure 2 It is the structural diagram of feature fusion module;

[0035] Figure 3 It is the overall network framework diagram;

[0036] Figure 4 This is a comparison chart of the visualization results. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0038] The present invention provides a three-dimensional point cloud super-resolution method based on multi-modal iterative fusion, comprising the following steps:

[0039] Step 1: Based on the 3D reconstructed point cloud dataset constructed by the manipulator and the DIGIT sensor in the Pybullet simulation environment, a point cloud super-resolution dataset 3DVT containing multiple tactile information is reconstructed.

[0040] The three-dimensional reconstructed point cloud data set includes real high-resolution point clouds and multiple tactile point clouds.

[0041] The point cloud super-resolution data set 3DVT includes high-resolution point clouds, low-resolution point clouds, and multiple tactile point clouds. The high-resolution point clouds and low-resolution point clouds are obtained by downsampling the real high-resolution point clouds.

[0042] The point cloud super-resolution dataset 3DVT contains a total of 12732 samples, including simple and complex objects, to verify the generalization ability of the model, 90% of which are taken as training sets and 10% as test sets.

[0043] Step 2, constructing a super-resolution network; the super-resolution network is used to fuse the input low-resolution point cloud and K-times tactile point clouds to output a high-resolution point cloud;

[0044] The super-resolution network includes a feature extraction module, a feature fusion module, a feature expansion module and a coordinate reconstruction module. The feature extraction module includes a tactile point cloud feature extraction module and a low-resolution point cloud feature extraction module.

[0045] Step 3: normalize the tactile point cloud and the low-score point cloud, and perform data enhancement. The data enhancement refers to rotating, scaling, randomly perturbing, and other operations on the normalized tactile point cloud and the low-score point cloud. By transforming and processing limited data, the model can learn better function expressions under richer training conditions, thereby showing better performance.

[0046] Step 4: Use the point cloud super-resolution dataset 3DVT containing multiple tactile information to train the super-resolution network, input the low-resolution point cloud (512 points) and the tactile point cloud (512 points) into the super-resolution network to obtain a high-resolution point cloud (8192 points).

[0047] Step 4.1, using a low-score feature extraction module and a tactile point cloud feature extraction module to perform preliminary feature extraction on the low-score point cloud and the K-times tactile point cloud, respectively, to obtain low-score point cloud features and tactile point cloud features;

[0048] Drawing on the idea of ​​Unet, the feature extraction module adopts an encoder-decoder structure, with skip connections between the encoder and the decoder. The encoder part is composed of convolutional layers, maximum pooling layers, etc., and the decoder adds deconvolution layers and skip connections.

[0049] Figure 1It is a tactile point cloud feature extraction module. The input is the tactile point cloud (512, 3). After two layers of convolution operations, the first intermediate feature of 512×16 is obtained. The maximum pooling layer downsamples it to 256×16. After two layers of convolution operations, the second intermediate feature of 256×32 is obtained. The maximum pooling layer downsamples it to 128×32. After two layers of convolution operations, the feature of 128×64 is obtained. The dimension is changed to 256×64 through the deconvolution layer. The skip connection operation is used to splice with the previously obtained second intermediate feature of 256×32, and then sent to the two-layer cascade convolution layer. The dimension is changed to 512×32 through the deconvolution layer, and then spliced ​​with the first intermediate feature of 512×16, and sent to the two-layer cascade convolution layer. Finally, the output feature (512, 4) is obtained.

[0050] The low-resolution point cloud feature extraction module is basically the same as the tactile point cloud feature extraction module. The difference is that the number of output feature channels of the low-resolution point cloud feature extraction is 16, and the number of output feature channels of the tactile point cloud feature extraction is 4, and the ratio between the two is 4:1.

[0051] In the feature extraction module, the input is a low-score point cloud (512 points) to obtain low-score point cloud features and tactile point cloud features 512×16, and the input is a tactile point cloud (N×512 points) to obtain tactile point cloud features N×512×4.

[0052] In step 4.2, the low-score point cloud features and K-times tactile features are sent to the feature fusion module for iterative fusion to obtain the overall features.

[0053] like Figure 2 As shown in the figure, the feature fusion module adopts a cascaded approach, including a splicing layer, a convolution layer, a maximum pooling layer, and a deconvolution layer. The fusion of multiple tactile features adopts a cyclic iterative approach.

[0054] When tactile features are first fused, the low-resolution point cloud features (512×16) and the first tactile point cloud features (512×4) are concatenated, and then sent to the convolution layer and the maximum pooling layer in sequence. In order to better retain the original low-resolution point cloud features, the processed features are concatenated with the original low-resolution point cloud features again and deconvoluted, and finally the first fused point cloud features (512×16) are obtained.

[0055] When adding multiple tactile senses, a loop iteration method is used. When performing the k+1th feature fusion, the kth fused features and the kth tactile point cloud features are used as inputs. After feature splicing, they are sequentially passed through the convolution layer and the maximum pooling layer, and then spliced ​​with the original low-resolution point cloud features, and then passed through the deconvolution layer to obtain the kth fused features. The above cycle is repeated until the Kth fused features are obtained.

[0056] The feature dimension of the point cloud after the Kth iteration fusion is 512 × 16. The value of k is (1, 2…K), K is the optimal number of touches, and through iteration, the super-resolution effect of the point cloud is observed in the experiment when the number of touches takes different values, so as to obtain the optimal value K in the current data set.

[0057] In step 4.3, the feature expansion module consists of a 5-layer cascaded Transformer encoder, which is used to extract high-dimensional features from the point cloud features after the K-th iterative fusion, and input them into the coordinate reconstruction module as expanded features; in the coordinate reconstruction module, the expanded features are first rearranged, and then mapped back to the three-dimensional coordinate space through a multi-layer perception mechanism to obtain a high-resolution point cloud.

[0058] In training, the chamfer distance CD is used as the loss function to minimize the distance between the low-resolution point cloud and the real high-resolution point cloud. The definition of the chamfer distance CD is shown in formula (1), where P represents the high-resolution point cloud predicted by the network, Q represents the high-resolution point cloud, p∈P represents the point in the predicted point cloud, and q∈Q represents the point in the high-resolution point cloud. The first term represents the sum of the minimum distances from any point p in P to Q, and the second term represents the sum of the minimum distances from any point q in Q to P. The larger the chamfer distance, the greater the difference between the two groups of point clouds P and Q. The smaller the chamfer distance, the closer the predicted point cloud is to the real point cloud, and the better the prediction result.

[0059]

[0060] The model uses the Pytorch deep learning framework, and performs preprocessing operations such as normalization and enhancement (rotation, scaling, random perturbation) on the point cloud. The Adam algorithm is used for optimization, with a learning rate of 0.001, an upsampling factor of 16, a batch size of 36, and 300 epochs trained on NVIDIA RTX A6000.

[0061] Step 5: Send the K tactile point clouds to be fused and the low-resolution point clouds into the trained super-resolution network to obtain high-resolution point clouds.

[0062] Table 1 Test results under different touch times

[0063]

[0064] Table 1 compares the experimental results under different touch times when the upsampling rate r=16. The input low-resolution point cloud consists of 512 points, each tactile point cloud consists of 512 points, and the output high-resolution point cloud consists of 8192 points. The evaluation indicator used is the chamfer distance CD. As can be seen from Table 1, the chamfer distance obtained by using tactile information assistance is smaller than that obtained without tactile information, and the test results are better. In addition, before using four touch information, CD decreases with the increase of the number of touches. As the number of touches increases, the rate of decrease of CD gradually slows down. When five touch information is used, CD increases, which may be due to the fact that when the number of touches is too high, the influence of tactile information on the overall features is too large. This result shows that there is an upper limit to the auxiliary effect of tactile information on visual super-resolution. In summary, combining tactile information can significantly improve the upsampling performance and the optimal value of the number of touches in this dataset is 4 times.

[0065] In order to more intuitively show the difference of high-resolution point clouds after adding tactile information, the trained model was used to perform qualitative analysis and visualization on the test set. Figure 4 As shown in the figure, three object models are selected for visualization and the local parts are enlarged for observation. From left to right, the three columns are joints, arches, and lamp posts. Figure 4 (a) in the figure is a high-resolution point cloud GT (8192 points). Figure 4 (b) in the figure is the input low-resolution point cloud (512 points) of the network, and the block area is a dense tactile point cloud, each tactile point cloud also contains 512 points. Figure 4 (c) shows the upsampling result without adding tactile information (8192 points). Figure 4 The upsampling result (8192 points) of adding four touch information is shown in (d). Comparing the high-resolution point cloud generated with and without tactile information, we can see Figure 4 Methods that do not add tactile information in (c) often introduce too much noise (e.g. Figure 4 (c) shows a partial enlarged view of the joint point cloud in Figure 2), clustering the points together in a non-uniform distribution (such as Figure 4 (c) in the partial enlarged view of the arch point cloud), or destroy some tiny structures in the result (such as Figure 4 (c) shows a partial enlarged view of the lamp post point cloud in Figure 1). In contrast, the method combining tactile information produces a visual result that is most similar to the real point cloud and can well preserve the uniformly distributed tiny local structures. It can be concluded that the addition of tactile information will not only affect the local effect when upsampling the point cloud, but also the tactile information will affect the upsampling result as part of the global information. The resulting high-resolution point cloud is more uniform in visual effect and has fewer local abnormal points.

[0066] In order to evaluate the effectiveness of the main components in this framework, ablation studies were performed on the feature extraction block and the feature fusion block. All models were trained and evaluated on the dataset 3DVT. The results in Table 2 show the effectiveness of the feature extraction module and the fusion module in integrating tactile information. The first row shows the experimental results of directly splicing the tactile point cloud without the feature extraction and feature fusion modules, and the degradation of upsampling quality can be seen. This is due to the destruction of the overall structure of the point cloud by dense local information, and the performance of the point cloud without tactile information is better than that of the point cloud with tactile information. The second row shows the experimental results of directly splicing the low-scoring point cloud features with the tactile features without the feature fusion module, and the third row shows the experimental results of the complete network framework. The table proves that the complete network framework always achieves the best performance with the lowest CD value at different numbers of touches. However, removing any component will lead to a decrease in overall performance, which means that each component in the framework contributes.

[0067] Table 2 Ablation experiment results

[0068]

Claims

1. A three-dimensional point cloud super-resolution method based on multi-modal iterative fusion, characterized in that: The steps include: Step 1: construct the point cloud super-resolution dataset 3DVT; The point cloud super-resolution dataset 3DVT includes high-resolution point clouds, low-resolution point clouds, and multiple tactile point clouds; Step 2: construct a super-resolution network, which is used to fuse the input low-resolution point cloud and multiple tactile point clouds to output a high-resolution point cloud; The super-resolution network includes a feature extraction module, a feature fusion module, a feature expansion module and a coordinate reconstruction module; The feature extraction module includes a tactile point cloud feature extraction module and a low-resolution point cloud feature extraction module; The low-resolution point cloud and the K-times tactile point cloud are respectively input into the low-resolution point cloud feature extraction module and the tactile point cloud feature extraction module, and feature extraction is performed on the feature point cloud to obtain low-resolution point cloud features and tactile point cloud features; the low-resolution point cloud features and the tactile point cloud features are input into the feature fusion module for feature fusion to obtain fused features, and then input into the feature expansion module, and the expanded features are input into the coordinate reconstruction module for reconstruction to obtain a high-resolution point cloud; The feature extraction module adopts an encoder-decoder structure, and a jump connection is used between the encoder and the decoder; the tactile point cloud feature extraction module and the low-score point cloud feature extraction module have the same structure, but different numbers of channels; The K times of tactile point clouds are input into the tactile point cloud feature extraction module to extract features from the feature point clouds. Specifically, each tactile point cloud is input into the tactile point cloud feature extraction module to obtain K tactile point cloud features, which are respectively recorded as k-th tactile point cloud features, wherein 1≤k≤K, and the steps are as follows: The kth tactile point cloud is subjected to two layers of convolution operations to obtain the first intermediate feature of the tactile point cloud. The maximum pooling layer downsamples the first intermediate feature of the tactile point cloud. After two layers of convolution operations, the second intermediate feature of the tactile point cloud is obtained. The maximum pooling layer downsamples the second intermediate feature of the tactile point cloud. After two layers of convolution operations and deconvolution operations, the second intermediate feature of the tactile point cloud is spliced ​​with the second intermediate feature of the tactile point cloud. The feature is then sent to two cascaded convolution layers. After the deconvolution layer, the feature is spliced ​​with the first intermediate feature of the tactile point cloud and sent to two cascaded convolution layers to obtain the kth tactile point cloud feature. The low-score point cloud features and the tactile point cloud features are input into a feature fusion module for feature fusion to obtain fused features. The specific steps are as follows: When K = 1; After the low-resolution point cloud features and the first tactile point cloud features are spliced, they are sent to the convolution layer and the maximum pooling layer in sequence, and the output features of the maximum pooling layer are spliced ​​with the low-resolution point cloud features again and a deconvolution operation is performed, and finally the first fused point cloud features are obtained; When K>1, the following steps are also included: The k-th fused features and the k-1th tactile point cloud features are used as input, and after feature splicing, they are passed through the convolution layer and the maximum pooling layer in sequence, and then spliced ​​with the original low-resolution point cloud features, and then passed through the deconvolution layer to obtain the k-th fused features, and the cycle is repeated until the K-th fused features are obtained. The K-th fused features are the fused features output by the feature extraction module; The feature expansion module is composed of a multi-layer cascaded Transformer encoder, which is used to extract high-dimensional features from the point cloud features after the K-th fusion, and input them into the coordinate reconstruction module as expanded features; in the coordinate reconstruction module, the expanded features are first rearranged to obtain rearranged features, and then the rearranged features are mapped back to the three-dimensional coordinate space through a multi-layer perception mechanism to obtain a high-resolution point cloud; Step 3, after preprocessing the low-resolution point cloud and K-times tactile point cloud in the point cloud super-resolution data set 3DVT, input them into the super-resolution network, and train the super-resolution network, wherein K is greater than or equal to 1; In the process of training the super-resolution network, the chamfer distance CD is used as the loss function to minimize the distance between the low-resolution point cloud and the high-resolution point cloud, as shown in the following formula: Where P represents the high-resolution point cloud predicted by the super-resolution network, Q represents the real high-resolution point cloud, p∈P represents the point in the predicted point cloud, q∈Q represents the point in the high-resolution point cloud in the dataset 3DVT, the first term represents the sum of the minimum distances from any point p in P to Q, and the second term represents the sum of the minimum distances from any point q in Q to P; Step 4: Send the K tactile point clouds to be fused and the low-resolution point clouds into the trained super-resolution network to obtain high-resolution point clouds.

2. The three-dimensional point cloud super-resolution method based on multi-modal iterative fusion according to claim 1, characterized in that: In step 3, the low-resolution point cloud and the K-times tactile point cloud in the point cloud super-resolution data set 3DVT are preprocessed, specifically, the tactile point cloud and the low-resolution point cloud are normalized, and data enhancement processing is performed.

3. The three-dimensional point cloud super-resolution method based on multi-modal iterative fusion according to claim 2, characterized in that: The data enhancement process includes rotating, scaling and randomly perturbing the normalized tactile point cloud and the low-score point cloud.

4. The three-dimensional point cloud super-resolution method based on multi-modal iterative fusion according to claim 1, characterized in that: K is the optimal number of touches. Through iteration, the super-resolution effect of the point cloud is observed in the experiment when the number of touches takes different values, so as to obtain the optimal number of touches K.

Images