Sparse point cloud completion method based on multi-view fusion

Abstract

The present invention belongs to the field of computer vision technology, and discloses a sparse point cloud completion method based on multi-view fusion, including: (1) inputting point cloud data, extracting point cloud features, and obtaining the overlapping area of ​​point cloud data; (2) calculating the similarity matrix of point clouds between frames in the overlapping area, and using the weighted singular value decomposition algorithm to solve the rigid transformation between frames; (3) realizing the precise registration of multi-view point clouds through the cumulative registration strategy; (4) using a 3D convolutional grid residual network module to perform coarse-grained completion of point clouds, and further using a multi-layer perceptron to fuse 3D voxel features and complete fine-grained accurate point cloud completion; (5) optimizing the model weights by calculating the loss function to improve the overall effect and accuracy of the completion. The present invention improves the accuracy and robustness of vehicle detection by integrating multi-view registration technology with a coarse-to-fine modular point cloud completion method.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and specifically relates to a sparse point cloud completion method based on multi-view fusion. Background Art

[0002] With the development of UGVs technology, its environmental perception and understanding capabilities have an important impact on the subsequent driving decision-making process. In the process of autonomous driving, the detection performance of surrounding vehicles is the key to ensuring the rationality of autonomous driving decisions. As a common environmental perception device, LiDAR is widely used in UGV technology. Its advantages are wide detection range and high detection accuracy, thus providing important basic data for vehicle detection. However, due to the inherent characteristics of LiDAR point cloud data such as sparsity and self-occlusion, the obtained vehicle point cloud structure is often incomplete, with only partial surface information. This problem limits the performance of UGVs in scene analysis, vehicle detection and other downstream tasks. Especially in tasks such as path planning and obstacle avoidance, the accuracy of vehicle detection is crucial. Therefore, improving the integrity of vehicle point clouds has become a research focus.

[0003] At present, deep learning models for point cloud completion are roughly divided into several types, including point convolution-based models, multi-view completion models, and generative adversarial network (GAN)-based models. Although these models have achieved satisfactory completion results in synthesizing dense point cloud datasets, the point cloud data collected by sensors in the real world is sparser than synthetic data. Existing models still have limitations and poor results when dealing with sparse point clouds and point cloud self-occlusion problems. In addition, the point clouds collected by unmanned vehicle technology are obtained from a series of continuous dynamic perspectives. Compared with static objects, the completion of mobile vehicles faces greater challenges.

[0004] At present, most mainstream completion models only use single-frame point cloud data, and few models can effectively process continuous frame point clouds from dynamic perspectives. Although existing models have attempted to consider the correlation of multi-view point clouds, they usually ignore the transformation correlation between continuous motion perspectives. Especially for vehicle detection tasks in autonomous driving scenarios, the focus is on completing the point clouds of key areas of the vehicle, rather than completing the point clouds of the global scene. As a moving target, the perspective of the vehicle changes significantly. The traditional single-frame completion model cannot fully capture this dynamic change, resulting in low quality of point cloud completion in key areas. Therefore, how to efficiently fuse continuous frame point clouds, especially the local key areas of the vehicle, has become a difficult problem that needs to be solved urgently.

[0005] In addition, point cloud data in real environments have problems with self-occlusion and surface incompleteness, and the lack of real labeled data as a training benchmark further increases the difficulty of the completion task. Most existing deep learning models rely on synthetic datasets for training, which usually have complete and ideal point cloud characteristics. However, there are large differences between synthetic datasets and real data in terms of point cloud density, noise, and occlusion. Although synthetic datasets can improve the performance of the completion model to some extent, the performance of the model often decreases significantly when tested in a real environment.

[0006] In summary, how to build an effective bridge between synthetic datasets and real data and improve the point cloud completion quality of the model in real scenarios is still a current research focus. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a sparse point cloud completion method based on multi-view fusion. After inputting the point cloud data, the completion effect of the sparse point cloud is optimized and improved by integrating the multi-view registration technology and the coarse-to-fine modular point cloud completion method. It is mainly used in the point cloud completion tasks and target detection tasks of the KITTI dataset to improve the accuracy and robustness of downstream vehicle detection.

[0008] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0009] The present invention discloses a sparse point cloud completion method based on multi-view fusion, which is implemented by a two-stage fusion network MVRC. The two-stage fusion network MVRC includes a multi-view cumulative registration module and a 3D convolutional grid residual network module. The multi-view cumulative registration module includes a feature extraction module, a preliminary registration module, a self-attention and cross-attention mechanism module, a feature calculation module and an optimization and output module. The input of the two-stage fusion network MVRC is an occlusion sequence. ,in , the predicted output of the multi-view cumulative registration module is Rigid transformation matrix And the accumulated multi-frame point cloud data after registration , the predicted output and true value of the 3D convolutional grid residual network module are the completed point clouds and real point cloud The multi-view cumulative registration module uses a 6-degree-of-freedom rigid transformation matrix Register the point clouds between adjacent frames.

[0010] The feature extraction module uses PointNet as a feature encoder to extract global features of each point cloud from each viewpoint. The PointNet network includes three MLPs, which extract features through 64-128 1×1 convolution kernels. PointNet is a deep learning-based network that encodes the features of the input point cloud by sharing weights and outputs the global features of the point cloud.

[0011] The preliminary registration module includes a feature fusion layer, a multi-layer MLP, a rotation matrix and translation vector layer, and a preliminary transformation layer. Based on the extracted global features, the MLP is used to calculate the initial transformation , and the point cloud data Convert to Point Cloud ;

[0012] The self-attention and cross-attention mechanism module includes 4 Transformer layers, each Transformer layer contains 8 attention heads, and uses the self-attention mechanism and the cross-attention mechanism to further optimize the alignment and extract the local features of the point cloud.

[0013] The feature calculation module includes a multi-layer MLP to calculate the point cloud data. and point cloud data Calculate the point-level features and output local features and ;

[0014] The optimization and output module includes a multi-layer MLP, which optimizes the registration matrix through the weighted singular value decomposition algorithm to reduce the registration error. Output the final registration matrix , used for subsequent point cloud transformation.

[0015] The sparse point cloud completion method comprises the following steps:

[0016] Step 1: Input point cloud data, extract point cloud features based on the Transformer mechanism, and obtain the overlapping area of ​​the point cloud data;

[0017] Step 2, calculate the similarity matrix of the point cloud between frames in the overlapping area obtained in step 1, and use the weighted singular value decomposition algorithm to solve the rigid transformation matrix between frames;

[0018] Step 3: The registration of multi-view point clouds is realized through the cumulative registration strategy of the two-stage fusion network MVRC, and the multi-frame point cloud data after registration is accumulated. ;

[0019] Step 4: Use 3D convolutional grid residual network module (GRNet) to perform point cloud data Perform coarse-grained completion and use a multi-layer perceptron to fuse 3D voxel features and complete fine-grained accurate point cloud completion;

[0020] Step 5: Optimize the weights of the 3D convolutional grid residual network module by calculating the loss function to improve the completion effect and accuracy.

[0021] A further improvement of the present invention is that the cumulative registration strategy in step 3 is specifically:

[0022] After calculating the rigid transformation matrix of the point cloud between frames After that, the point clouds of multiple adjacent frames are gradually registered to the same reference frame. For the point cloud data obtained from the continuous perspective of the vehicle in autonomous driving, the cumulative transformation matrix between frames is defined as , from the frame To Frame The registration transformation matrix is ​​accumulated by matrix multiplication, and the formula is as follows:

[0023]

[0024] in, From the frame To Frame The transformation matrix, From the frame To Frame The transformation matrix, From the frame To Frame The transformation moment, From the frame To Frame The transformation matrix of

[0025] According to the cumulative registration strategy, the point cloud data is accurately converted to the reference frame , to achieve accurate alignment of multi-frame point clouds, the accumulated multi-frame point cloud data after registration is expressed as ,in is the number of points, , and used as the input of the next two-stage fusion network MVRC to complete the sparse point cloud data.

[0026] A further improvement of the present invention is that: Step 1 inputs point cloud data, extracts point cloud features based on the Transformer mechanism, and obtains the overlapping area of ​​the point cloud data, specifically including the following steps:

[0027] Step 1.1: Input point cloud data of adjacent frames of KITTI dataset and , using weight-shared PointNet as the feature encoder, extracting the global features of each frame of point cloud, mixing the extracted global features, and using a multi-layer perceptron to calculate the initial transformation matrix ;

[0028] Step 1.2: Point cloud data According to the initial transformation Perform rough alignment to obtain the transformed point cloud ;

[0029] Step 1.3: Extract each point cloud using self-attention and cross-attention mechanisms Local features and , and through the similarity matrix Calculate the matching degree between adjacent frames and determine the overlapping area of ​​the point cloud data. The formula is as follows:

[0030]

[0031] in, For point cloud data The local features in For point cloud data The local features in is the attention matrix, is the query matrix, is a numerical matrix, For point cloud data The numerical matrix in For point cloud data The transpose of the key matrix in , For point cloud data The transpose of the key matrix in , For point cloud data The Point-wise features of the layer, For point cloud data Middle Point-wise features of the layer, is the weight matrix of the query matrix, is the weight matrix of the key matrix, is a weight matrix of numeric matrices.

[0032] A further improvement of the present invention is that: Step 2 calculates the similarity matrix of the point clouds between frames in the overlapped area, and uses a weighted singular value decomposition algorithm to solve the rigid transformation between frames, which specifically includes the following steps:

[0033] Step 2.1: Filter out to obtain high matching point pairs, and use relaxed constraints to define the matching value of each point pair , ensuring that each point can be paired with at most one other point;

[0034] Step 2.2: From the similarity matrix Obtained before The corresponding relationship , the top corresponding point pair set extracted from the similarity matrix , using the weighted singular value decomposition algorithm to calculate the rigid transformation matrix of the point cloud between frames ,The point clouds between adjacent frames are aligned, and the point clouds between adjacent frames are accurately registered in the overlapping area.

[0035] A further improvement of the present invention is that step 4 specifically includes the following steps:

[0036] Step 4.1: Accumulate and register multiple point cloud data After that, due to the number of points Relatively large, uniform downsampling method is used to remove redundant points;

[0037] Step 4.2: The 3D convolutional grid residual network uses skip connections to perform point cloud data in the coarse completion stage. Processing, unstructured point cloud data is converted to a regular 3D grid , from the 3D mesh Convert to a rough point cloud ;

[0038] Step 4.2: Convert the unstructured point cloud into a regular 3D grid. is converted to a regular 3D grid , in the rough completion stage, the point cloud data The 3D convolutional grid residual network combines 4 groups of 3D convolutional layers, 4 groups of 3D deconvolutional layers and 2 groups of fully connected layers through skip connections, from the 3D grid Convert to a rough point cloud .

[0039] Step 4.3: Use a multi-layer perceptron to analyze the rough point cloud of step 4.2 The residuals are completed, and the features in all 3D convolutional grid residual networks are aggregated into a grid feature vector , where q is the number of grids in the 3D point cloud, and the network feature vector is selected The point cloud is completed through 4 layers of MLP to capture the detailed information of the point cloud. The multi-layer perceptron fits the coordinate residual between the rough point cloud and the real point cloud, and finally generates a finely completed point cloud. , and according to the rigid transformation matrix The weights and parameters of the two-stage fusion network MVRC are optimized to minimize the gap between the predicted value and the true value.

[0040] A further improvement of the present invention is that the loss function in step 5 is:

[0041] Registration loss for multi-view point clouds :

[0042]

[0043] Distance loss for point cloud fusion between frames :

[0044]

[0045] Mesh loss in the coarse-grained completion phase :

[0046]

[0047] Point cloud loss in fine-grained completion stage :

[0048]

[0049] in, is the real rotation matrix, is the predicted rotation matrix, is the true translation vector, The predicted translation vector, q is the number of grids in the 3D point cloud, To complete the grid, For a real grid, To complete the point cloud, It is a real point cloud.

[0050] Registration loss Compute the predicted rotation matrix with the predicted translation vector and the real rotation matrix With the real translation vector The difference between; fusion loss By comparison The distance between the points before and after the frame registration is used for measurement. In the completion phase, the grid loss Measuring the difference of gridded point clouds, fine-grained completion loss Completing point cloud by chamfer distance calculation With real point cloud By back-propagating the loss, the weights of the two-stage fusion network MVRC are optimized to improve the completion accuracy.

[0051] The beneficial effects of the present invention are:

[0052] 1. This paper proposes a two-stage fusion model MVRC to realize the registration and completion of multi-frame point clouds, aiming to improve the vehicle detection performance of unmanned ground vehicles (UGVs) in motion.

[0053] 2. The present invention designs a cumulative registration strategy, through which rigid registration of multiple local frames is performed.

[0054] 3. The coarse-to-fine residual completion module adopted in the present invention helps to complete the registration result, thereby improving the vehicle detection performance on the KITTI dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of the sparse point cloud completion method based on multi-view fusion of the present invention.

[0056] Figure 2 This is a schematic diagram of a sparse point cloud completion model based on multi-view fusion, namely, the MVRC model.

[0057] Figure 3 It is a schematic diagram of the visualization result of target detection of the present invention. DETAILED DESCRIPTION

[0058] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0059] like Figure 1 and Figure 2 As shown, the present invention is a sparse point cloud completion method based on multi-view fusion, the sparse point cloud completion method is implemented by a two-stage fusion network MVRC, the two-stage fusion network MVRC includes a multi-view cumulative registration module and a 3D convolutional grid residual network module, the multi-view cumulative registration module includes a feature extraction module, a preliminary registration module, a self-attention and cross-attention mechanism module, a feature calculation module and an optimization and output module, the input of the two-stage fusion network MVRC is an occlusion sequence ,in , the predicted output of the multi-view cumulative registration module is Rigid transformation matrix And the accumulated multi-frame point cloud data after registration , the predicted output and true value of the 3D convolutional grid residual network module are the completed point clouds and real point cloud The multi-view cumulative registration module uses a 6-degree-of-freedom rigid transformation matrix The point clouds between adjacent frames are registered, wherein the feature extraction module uses PointNet as a feature encoder to extract the global features of each viewpoint point cloud. The PointNet network includes 3 MLPs, which extract features through 64-128 1×1 convolution kernels. PointNet is a network based on deep learning. It encodes the features of the input point cloud by sharing weights and outputs the global features of the point cloud. The preliminary registration module includes a feature fusion layer, a multi-layer MLP, a rotation matrix and translation vector layer, and a preliminary transformation layer. Based on the extracted global features, the MLP is used to calculate the initial transformation , and the point cloud data Convert to Point Cloud The self-attention and cross-attention mechanism module includes 4 Transformer layers, each Transformer layer contains 8 attention heads, and uses the self-attention mechanism and cross-attention mechanism to further optimize the registration and extract the local features of the point cloud. The feature calculation module includes a multi-layer MLP to perform point cloud data and point cloud data Calculate the point-level features and output local features and The optimization and output module includes a multi-layer MLP, which optimizes the registration matrix through the weighted singular value decomposition algorithm to reduce the registration error. Output the final registration matrix , used for subsequent point cloud transformation.

[0060] like Figure 1 As shown, the sparse point cloud completion method of the present invention includes the following steps:

[0061] Step 1: Input point cloud data, extract point cloud features based on the Transformer mechanism, and obtain the overlapping area of ​​the point cloud data. The specific steps include the following:

[0062] Step 1.1: Input point cloud data of adjacent frames of KITTI dataset and , using weight-shared PointNet as the feature encoder, extracting the global features of each frame of point cloud, mixing the extracted global features, and using a multi-layer perceptron to calculate the initial transformation matrix ;

[0063] Step 1.2: Point cloud data According to the initial transformation Perform rough alignment to obtain the transformed point cloud ;

[0064] Step 1.3: Extract each point cloud using self-attention and cross-attention mechanisms Local features and , and through the similarity matrix Calculate the matching degree between adjacent frames and determine the overlapping area of ​​the point cloud data. The formula is as follows:

[0065]

[0066] in, For point cloud data The local features in For point cloud data The local features in is the attention matrix, is the query matrix, is a numerical matrix, For point cloud data The numerical matrix in For point cloud data The transpose of the key matrix in , For point cloud data The transpose of the key matrix in , For point cloud data The Point-wise features of the layer, For point cloud data Middle Point-wise features of the layer, is the weight matrix of the query matrix, is the weight matrix of the key matrix, is a weight matrix of numeric matrices.

[0067] Step 2: Calculate the similarity matrix of the point cloud between frames in the overlapping area obtained in step 1, and use the weighted singular value decomposition algorithm to solve the rigid transformation matrix between frames, which specifically includes the following steps:

[0068] Step 2.1: Filter out to obtain high matching point pairs, and use relaxed constraints to define the matching value of each point pair , ensuring that each point can be paired with at most one other point;

[0069] Step 2.2: From the similarity matrix Obtained before The corresponding relationship , the top corresponding point pair set extracted from the similarity matrix , using the weighted singular value decomposition algorithm to calculate the rigid transformation matrix of the point cloud between frames ,The point clouds between adjacent frames are aligned, and the point clouds between adjacent frames are accurately registered in the overlapping area.

[0070] Step 3: The registration of multi-view point clouds is realized through the cumulative registration strategy of the two-stage fusion network MVRC, and the multi-frame point cloud data after registration is accumulated. The cumulative registration strategy in this step is as follows:

[0071] After calculating the rigid transformation matrix of the point cloud between frames After that, the point clouds of multiple adjacent frames are gradually registered to the same reference frame. For the point cloud data obtained from the continuous perspective of the vehicle in autonomous driving, the cumulative transformation matrix between frames is defined as , from the frame To Frame The registration transformation matrix is ​​accumulated by matrix multiplication, and the formula is as follows:

[0072]

[0073] in, From the frame To Frame The transformation matrix, From the frame To Frame The transformation matrix, From the frame To Frame The transformation moment, From the frame To Frame The transformation matrix of

[0074] According to the cumulative registration strategy, the point cloud data is accurately converted to the reference frame , to achieve accurate alignment of multi-frame point clouds, the accumulated multi-frame point cloud data after registration is expressed as ,in is the number of points, , and used as the input of the next two-stage fusion network MVRC to complete the sparse point cloud data.

[0075] Step 4: Use 3D convolutional grid residual network module (GRNet) to perform point cloud data Perform coarse-grained completion, and use a multi-layer perceptron to fuse 3D voxel features and complete fine-grained accurate point cloud completion; specifically, the following steps are included:

[0076] Step 4.1: Accumulate and register multiple point cloud data After that, due to the number of points Relatively large, uniform downsampling method is used to remove redundant points;

[0077] Step 4.2: The 3D convolutional grid residual network uses skip connections to perform point cloud data in the coarse completion stage. Processing, unstructured point cloud data is converted to a regular 3D grid , in the rough completion stage, the point cloud data The 3D convolutional grid residual network combines 4 groups of 3D convolutional layers, 4 groups of 3D deconvolutional layers and 2 groups of fully connected layers through skip connections, from the 3D grid Convert to a rough point cloud ;

[0078] Step 4.3: Use a multi-layer perceptron to analyze the rough point cloud of step 4.2 The residuals are completed, and the features in all 3D convolutional grid residual networks are aggregated into a grid feature vector , where q is the number of grids in the 3D point cloud, and the network feature vector is selected The point cloud is completed through 4 layers of MLP to capture the detailed information of the point cloud. The multi-layer perceptron fits the coordinate residual between the rough point cloud and the real point cloud, and finally generates a finely completed point cloud. , and according to the rigid transformation matrix The weights and parameters of the two-stage fusion network MVRC are optimized to minimize the gap between the predicted value and the true value.

[0079] Step 5: Optimize the weights of the 3D convolutional grid residual network module by calculating the loss function to improve the completion effect and accuracy. The loss function in this step is:

[0080] Registration loss for multi-view point clouds :

[0081]

[0082] Distance loss for point cloud fusion between frames :

[0083]

[0084] Mesh loss in the coarse-grained completion phase :

[0085]

[0086] Point cloud loss in fine-grained completion stage :

[0087]

[0088] in, is the real rotation matrix, is the predicted rotation matrix, is the true translation vector, The predicted translation vector, q is the number of grids in the 3D point cloud, To complete the grid, For a real grid, To complete the point cloud, It is a real point cloud.

[0089] Registration loss Compute the predicted rotation matrix with the predicted translation vector and the real rotation matrix With the real translation vector The difference between; fusion loss By comparison The distance between the points before and after the frame registration is used for measurement. In the completion phase, the grid loss Measuring the difference of gridded point clouds, fine-grained completion loss Completing point cloud by chamfer distance calculation With real point cloud By back-propagating the loss, the weights of the two-stage fusion network MVRC are optimized to improve the completion accuracy.

[0090] The present invention quantitatively analyzes the experimental results of multi-view point cloud fusion through the performance of vehicle detection. The target task samples in the KITTI dataset were initially divided into 7481 training sets and 7518 test sets. Since the true values ​​of the test sets are unavailable and access to the test server is limited, for experimental research, the present invention divides the training set into 5241 training sets and 2240 validation sets, which contain 2401 vehicles. In order to make the samples of the training set and the validation set more diversified, samples from different sequences are selected as much as possible in the training set and the validation set to evaluate the performance of multi-view point cloud fusion. In addition, the present invention also uses the first 50 frames in the validation set, which contain 31 vehicles, to verify the effectiveness of each part of the proposed two-stage fusion network MVRC.

[0091] In order to verify the effect of the two-stage fusion network MVRC, this paper selects several classic point cloud completion models for comparative experiments, including 3DCapsule, PCN, PFNet, GRNet and PoinTr. The minimum matching distance (MMD), fidelity distance (FD) and consistency (Consistency) are used as evaluation indicators, and the experimental results are shown in Table 1.

[0092] Table 1

[0093]

[0094] Subsequently, the present invention puts the completion result of the two-stage fusion network MVRC into the target detection model PointPillar, and compares the effect of vehicle target detection before and after completion. mIoU, mAOS, and mAP are selected, and the IoU thresholds are 0.5 and 0.75 respectively as the evaluation indicators. The experimental data are shown in Table 2. The target detection visualization results are shown in Figure 3 .

[0095] Table 2

[0096]

[0097] The present invention can realize shape completion of point clouds collected from continuously moving perspectives. At the same time, the present invention also creates a new training data set specifically for training the proposed two-stage fusion network MVRC, focusing on solving the registration problem of vehicle point clouds, aiming to improve the effect of vehicle detection in autonomous driving scenarios, and improving the accuracy and robustness of vehicle detection by integrating multi-perspective registration technology with a coarse-to-fine modular point cloud completion method.

[0098] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A sparse point cloud completion method based on multi-view fusion, characterized by: The sparse point cloud completion method is implemented by a two-stage fusion network MVRC, which includes a multi-view cumulative registration module and a 3D convolutional grid residual network module. The multi-view cumulative registration module includes a feature extraction module, a preliminary registration module, a self-attention and cross-attention mechanism module, a feature calculation module, and an optimization and output module. Specifically, the sparse point cloud completion method includes the following steps: Step 1: Input point cloud data, extract point cloud features based on the self-attention and cross-attention mechanism module Transformer layer, and obtain the overlapping area of ​​the point cloud data; Step 2, calculate the similarity matrix of the point cloud between frames in the overlapping area obtained in step 1, and use the weighted singular value decomposition algorithm to solve the rigid transformation matrix between frames; Step 3: The registration of multi-view point clouds is realized through the cumulative registration strategy of the two-stage fusion network MVRC, and the multi-frame point cloud data after registration is accumulated. ; Step 4: Use 3D convolutional grid residual network module to train point cloud data Perform coarse-grained completion and use a multi-layer perceptron to fuse 3D voxel features and complete fine-grained accurate point cloud completion; Step 5: Optimize the weights of the 3D convolutional grid residual network module by calculating the loss function to improve the completion effect and accuracy.

2. The sparse point cloud completion method based on multi-view fusion according to claim 1, characterized in that: The input of the two-stage fusion network MVRC is the occlusion sequence ,in , the predicted output of the multi-view cumulative registration module is Rigid transformation matrix And the accumulated multi-frame point cloud data after registration , the predicted output and true value of the 3D convolutional grid residual network module are the completed point clouds and real point cloud The multi-view cumulative registration module uses a rigid transformation matrix Register the point clouds between adjacent frames. Among them, the feature extraction module uses PointNet as a feature encoder to extract the global features of each viewpoint point cloud. The PointNet network includes three multi-layer perceptrons; The preliminary registration module includes a feature fusion layer, a multi-layer perceptron, a rotation matrix and translation vector layer, and a preliminary transformation layer. The multi-layer perceptron is used to calculate the initial transformation. , and the point cloud data Convert to Point Cloud ; The self-attention and cross-attention mechanism module includes 4 Transformer layers, each of which contains 8 attention heads; The feature calculation module includes a multi-layer perceptron to calculate the point cloud data. and point cloud data Calculate the point-level features and output local features and ; The optimization and output module includes a multi-layer perceptron, which optimizes the initial transformation through a weighted singular value decomposition algorithm. , reduce the registration error and output the final registration matrix , used for subsequent point cloud transformation.

3. The sparse point cloud completion method based on multi-view fusion according to claim 1, characterized in that: The cumulative registration strategy in step 3 is as follows: After calculating the rigid transformation matrix of the point cloud between frames After that, the point clouds of multiple adjacent frames are registered to the same reference frame. For the point cloud data obtained from the continuous perspective of the vehicle in autonomous driving, the cumulative transformation matrix between frames is defined as , from the frame To Frame The registration transformation matrix is ​​accumulated by matrix multiplication, and the formula is as follows: ， in, From the frame To Frame The transformation matrix, From the frame To Frame The transformation matrix, From the frame To Frame The transformation moment, From the frame To Frame The transformation matrix of According to the cumulative registration strategy, the point cloud data is accurately converted to the reference frame , to achieve accurate alignment of multi-frame point clouds, the accumulated multi-frame point cloud data after registration is expressed as ,in is the number of points, , and used as the input of the next two-stage fusion network MVRC to complete the sparse point cloud data.

4. The sparse point cloud completion method based on multi-view fusion according to any one of claims 1 to 3, characterized in that: Step 1: Input point cloud data, extract point cloud features based on the Transformer mechanism, and obtain the overlapping area of ​​point cloud data. The specific steps include: Step 1.1: Input point cloud data of adjacent frames of KITTI dataset and , using weight-shared PointNet as the feature encoder, extracting the global features of each frame of point cloud, mixing the extracted global features, and using a multi-layer perceptron to calculate the initial transformation matrix ; Step 1.2: Point cloud data According to the initial transformation Perform rough alignment to obtain the transformed point cloud ; Step 1.3: Extract each point cloud using self-attention and cross-attention mechanisms Local features and , and through the similarity matrix Calculate the matching degree between adjacent frames and determine the overlapping area of ​​the point cloud data. The formula is as follows: ， in, For point cloud data The local features in For point cloud data The local features in is the attention matrix, is the query matrix, is a numerical matrix, For point cloud data The numerical matrix in For point cloud data The transpose of the key matrix in , For point cloud data The transpose of the key matrix in , For point cloud data The Point-wise features of the layer, For point cloud data Middle Point-wise features of the layer, is the weight matrix of the query matrix, is the weight matrix of the key matrix, is a weight matrix of numeric matrices.

5. The method for completing sparse point cloud based on multi-view fusion according to claim 4, characterized in that: Step 2 calculates the similarity matrix of the point clouds between frames in the overlapped area, and uses the weighted singular value decomposition algorithm to solve the rigid transformation between frames, which specifically includes the following steps: Step 2.1: Filter out the similarity matrix to obtain high matching point pairs, and use relaxed constraints to define the matching value of each point pair , ensuring that each point can be paired with at most one other point; Step 2.2: From the similarity matrix Obtained before The corresponding relationship , the top corresponding point pair set extracted from the similarity matrix , using the weighted singular value decomposition algorithm to calculate the rigid transformation matrix of the point cloud between frames ,The point clouds between adjacent frames are aligned, and the point clouds between adjacent frames are accurately registered in the overlapping area.

6. The sparse point cloud completion method based on multi-view fusion according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4.1: Accumulate and register multiple point cloud data Finally, a uniform downsampling method is used to remove redundant points; Step 4.2: The 3D convolutional grid residual network uses skip connections to perform point cloud data in the coarse completion stage. Processing, unstructured point cloud data is converted to a regular 3D grid , from the 3D mesh Convert to a rough point cloud ; Step 4.3: Use a multi-layer perceptron to analyze the rough point cloud of step 4.2 The residuals are completed, and the features in all 3D convolutional grid residual networks are aggregated into a grid feature vector , where q is the number of grids in the 3D point cloud, and the network feature vector is selected The point cloud is completed through 4 layers of MLP to capture the detailed information of the point cloud. The multi-layer perceptron fits the coordinate residual between the rough point cloud and the real point cloud, and finally generates a finely completed point cloud. , and according to the rigid transformation matrix The weights and parameters of the two-stage fusion network MVRC are optimized to minimize the gap between the predicted value and the true value.

7. The method for completing sparse point cloud based on multi-view fusion according to claim 1, characterized in that: The loss function in step 5 is: Registration loss for multi-view point clouds : ， Distance loss for point cloud fusion between frames : ， Mesh loss in the coarse-grained completion phase : ， Point cloud loss in fine-grained completion stage : ， in, is the real rotation matrix, is the predicted rotation matrix, is the true translation vector, The predicted translation vector, q is the number of grids in the 3D point cloud, To complete the grid, For a real grid, To complete the point cloud, is the real point cloud; Registration loss Compute the predicted rotation matrix with the predicted translation vector and the real rotation matrix With the real translation vector The difference between; fusion loss By comparison The distance between the points before and after the frame registration is used for measurement. In the completion phase, the grid loss Measuring the difference of gridded point clouds, fine-grained completion loss Completing point cloud by chamfer distance calculation With real point cloud By back-propagating the loss, the weights of the two-stage fusion network MVRC are optimized to improve the completion accuracy.

Images