Multi-view point cloud registration method based on deep learning
By employing a deep learning-based multi-view point cloud registration method, which utilizes PointNet and Transformer architectures for feature extraction and registration, the problem of long computation time and large cumulative error in multi-view point cloud registration is solved, achieving efficient and robust point cloud registration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2026-03-10
AI Technical Summary
Multi-view point cloud registration takes a long time and has a large cumulative error. Existing methods are inefficient and prone to getting trapped in local optima when the amount of computation increases.
A deep learning-based approach is adopted, which extracts point cloud features through the PointNet network, performs point cloud registration using the Transformer structure and registration matrix calculation module, and combines feature fusion and singular value decomposition to achieve an end-to-end registration process.
It improves the robustness and computation speed of point cloud registration, reduces cumulative errors, has better adaptability, and improves computational efficiency.
Smart Images
Figure CN121639749A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine vision, specifically a multi-view point cloud registration method based on deep learning. Background Technology
[0002] With the rapid development of 3D measurement technology, 3D data of objects or environments can be acquired quickly and conveniently. However, due to the limited field of view of sensors, a complete model of the object or environment cannot be obtained. Therefore, it is necessary to transform point cloud data from various viewpoints to a unified coordinate system to form a complete point cloud model. The process of transformation is called point cloud registration. Point cloud registration is essentially the estimation of rotation and translation parameters between point clouds. It is a fundamental task in computer vision and has wide applications in fields such as 3D object measurement, autonomous driving, and virtual reality.
[0003] Depending on the number of point clouds to be registered, point cloud registration is divided into pairwise view point cloud registration and multi-view point cloud registration. Pairwise view point cloud registration involves registering two sets of point clouds with a certain overlap area to a single coordinate system. In contrast, multi-view point cloud registration is more difficult than pairwise view point cloud registration due to the complexity of the relationships between point clouds and the existence of accumulated errors. Some methods propose strategies that utilize ICP (Iterative Closet Point) to successively register and merge two point cloud datasets to ultimately complete the model. However, as the amount of data increases, the ICP algorithm increases computation time and suffers from the problem of accumulated errors. Other methods treat the point cloud as coming from a single Gaussian mixture model and use the expectation-maximization algorithm to estimate the parameters of the Gaussian mixture model and register the optimal rigid transformation. However, these algorithms require estimating many Gaussian mixture model parameters and are prone to getting trapped in local optima. Summary of the Invention
[0004] This invention provides a multi-view point cloud registration method based on deep learning to overcome the problems of long calculation time and high cumulative registration error in multi-view point cloud registration.
[0005] The technical solution adopted by this invention to achieve the above objectives is: a multi-view point cloud registration method based on deep learning, comprising the following steps:
[0006] Step 1: Sample the point cloud set to be registered from the multi-view data to form the point cloud set to be registered.
[0007] Step 2: For the elements in the point cloud set to be registered, use the feature extraction method to extract and represent the features of each point in the element, and obtain the feature vector corresponding to each point. Combine the feature vectors corresponding to different points to form the feature set of each point cloud.
[0008] Step 3: The feature sets of each point cloud are used as input, and the registration matrix is calculated through the registration network;
[0009] Step 4: Register the point clouds to be registered according to the registration matrix, unify the point clouds to be registered to the same coordinate system, and generate a complete point cloud model of the object.
[0010] The point cloud to be registered is sampled using a key point sampling method; where key points are defined as local extrema or corner points, and the sampling method is implemented through gradient operators.
[0011] The feature extraction method includes the following steps:
[0012] A pre-trained point cloud feature extraction network is used; the initial network parameters are obtained through transfer learning, and then the final network parameters are obtained by training on the task target dataset.
[0013] The point cloud feature extraction network uses the PointNet network; the extracted feature set obtained from the point cloud to be registered through the PointNet network is FSi = {f j ,j∈N}, where f j N represents the extracted features of a point in the point cloud to be registered, where N represents the number of features.
[0014] The registration network consists of a Transformer structure and a registration matrix calculation module. The Transformer structure is used to fuse the features of each point cloud to be registered with the features of the other point clouds to be registered, so as to introduce global information in the point cloud registration process. The registration matrix calculation module is used to calculate the registration matrix.
[0015] The Transformer structure takes the extracted feature set FSi of the i-th point cloud to be registered as the input of the Transformer structure, and the extracted feature set FSi_others of other point clouds as the reference output. The residual φ(FSi,FSi_others) is calculated, and then the fused feature set FSi+=FSi+φ(FSi,FSi_others) corresponding to FSi is obtained.
[0016] The registration matrix calculation module performs the following steps:
[0017] Pair point generation: For a point x in the extracted feature set FSi j According to the formula Calculation; where Y is the set of extracted features from other point clouds, FSi_others. To calculate x using the inner product j The softmax() function is used to calculate the similarity between points in Y and x. jThe matching probability with each point of Y, Calculate x j The corresponding average point, For x j Adjusted target point;
[0018] Registration parameter calculation: Adjust the target points Singular value decomposition is used to obtain the registration matrix.
[0019] The registration network is obtained through training; the point cloud feature set input through the registration matrix calculation module is used to compare the obtained registration matrix with the ground truth, calculate the loss function and backpropagate it to the registration network to optimize and obtain the parameters of the registration network.
[0020] A deep learning-based multi-view point cloud registration system includes:
[0021] The point cloud acquisition unit is used to sample the point cloud set to be registered from multiple views to form the point cloud set to be registered.
[0022] The feature extraction unit is used to extract and represent features of each point in the element set of point clouds to be registered using the feature extraction method, so as to obtain the feature vector corresponding to each point. The feature vectors corresponding to different points are combined to form the feature set of each point cloud.
[0023] The registration network takes the feature sets of each point cloud as input and calculates the registration matrix through the registration network;
[0024] The multi-view point cloud registration unit registers the point clouds to be registered according to the registration matrix, unifying each point cloud to be registered into the same coordinate system, and generating a complete point cloud model of the object.
[0025] The registration network includes:
[0026] The extracted feature set FSi of the i-th point cloud to be registered is used as the input of the Transformer structure. After passing through the first multi-head attention, it is processed by layer normalization and summed with FSi through the first residual connection to obtain the latent feature FH1. After passing through the feedforward layer, FH1 is processed by normalization and summed with FH1 through the second residual connection to obtain the latent feature FH2.
[0027] The extracted feature set FSi_others from other point clouds to be registered is concatenated through a combination layer to obtain latent feature FH3. After passing through the second multi-head attention and undergoing layer normalization, it is summed with FH3 through the third residual connection to obtain latent feature FH4. FH2 and FH4 are input into the third multi-head attention, processed through a normalization layer, and summed with FH4 through the fourth residual connection to obtain latent feature FH5. FH5 passes through the second feedforward network, undergoes layer normalization, and is summed with FH5 through the second residual connection to obtain latent feature FH6. FH6 is then summed with FSi through a residual connection layer to obtain the fused feature set FSi+.
[0028] The present invention has the following beneficial effects and advantages:
[0029] 1. This invention uses deep learning methods to extract features from point clouds and uses them for registration. It is insensitive to noise and outliers, has wide adaptability, and better robustness.
[0030] 2. This invention uses a deep network based on an attention mechanism to calculate registration parameters, thereby reducing accumulated errors by obtaining the optimal registration sequence.
[0031] 3. This invention employs an end-to-end registration method, which reduces the size of the search space and increases the computation speed. Attached Figure Description
[0032] Figure 1 A flowchart illustrating the deep learning-based multi-view point cloud registration method provided in this invention.
[0033] The components include: 1. Point cloud to be registered; 2. Feature extraction module; 3. Registration network; 3.1 Transformer structure; 3.2 Registration matrix calculation; 4. Multi-view point cloud registration; 5. Loss function.
[0034] Figure 2 A detailed structural diagram of the deep learning-based multi-view point cloud registration method provided in this invention;
[0035] Figure 3 The feature extraction network structure provided by this invention;
[0036] Figure 4 The Transformer structure diagram provided by this invention. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] like Figure 1 As shown, a multi-view point cloud registration method based on deep learning includes the following steps:
[0039] Step 1: Sample the point cloud set that needs to be registered to form a point cloud set to be registered;
[0040] Step 2: For each element in the point cloud set to be registered, feature extraction methods are used to extract and represent features of each point in the element, resulting in a feature vector corresponding to each point. The feature vectors corresponding to different points are combined to form the feature set of each point cloud.
[0041] Step 3: Using the feature sets of each point cloud from Step 2 as input, calculate the registration matrix through the registration network;
[0042] Step 4: Register the point cloud according to the registration matrix in Step 3, unify the point clouds to the same coordinate system, and generate a complete point cloud model of the object.
[0043] In this embodiment, the method for sampling the point cloud set to be registered in step one is a key point sampling method, where key points are defined as local extrema or corner points. The sampling method is implemented through gradient operators. Preferably, the SIFT operator is used for local extrema extraction and the Harris operator is used for corner point extraction.
[0044] In this embodiment, as Figure 2 As shown, the feature extraction method described in step two is a pre-trained point cloud feature extraction network. The coarse network parameters are obtained through transfer learning, and then fine-tuned using the target dataset to obtain accurate network parameters. Preferably, the point cloud feature extraction network can use the PointNet network. PointNet solves the problem of directly processing 3D point clouds using 3D convolutional neural networks. It can extract point set features even under adverse conditions such as point cloud noise and outliers, exhibiting excellent robustness. The PointNet network structure is as follows: Figure 3 As shown.
[0045] like Figure 2As shown, in this embodiment, the registration network consists of a Transformer structure and a registration matrix calculation module. The Transformer structure is used to fuse the features of each point cloud with the features of the other point clouds, so as to introduce global information in the point cloud registration process. The registration matrix calculation module is used to calculate the registration parameters. Preferably, as follows... Figure 4 As shown, let the feature set obtained from a point cloud through PointNet be denoted as FSi = {f j ,j∈N}, where f j FSi is the feature extracted from a point in the point cloud. Taking FSi as the input of the Transformer structure, and introducing the other point cloud feature set FSi_others as the reference output, the residual φ(FSi,FSi_others) is calculated. Then, the fused feature set corresponding to FSi is FSi+=FSi+φ(FSi,FSi_others).
[0046] like Figure 4 As shown, in the registration network, the extracted feature set FSi of the i-th point cloud to be registered is used as the input of the Transformer structure. After passing through the first multi-head attention, it is processed by layer normalization and summed with FSi through the first residual connection to obtain the latent feature FH1. After passing through the feedforward layer, FH1 is normalized and summed with FH1 through the second residual connection to obtain the latent feature FH2.
[0047] The extracted feature set FSi_others from other point clouds to be registered is concatenated through a combination layer to obtain latent feature FH3. After passing through the second multi-head attention and undergoing layer normalization, it is summed with FH3 through the third residual connection to obtain latent feature FH4. FH2 and FH4 are input into the third multi-head attention, processed through a normalization layer, and summed with FH4 through the fourth residual connection to obtain latent feature FH5. FH5 passes through the second feedforward network, undergoes layer normalization, and is summed with FH5 through the second residual connection to obtain latent feature FH6. FH6 is then summed with FSi through a residual connection layer to obtain the fused feature set FSi+.
[0048] like Figure 2 As shown, the registration calculation module in this embodiment consists of pairing point generation and registration parameter calculation. The method for generating pairing points is as follows: for a point x in FSi... j According to the formula Calculate, where Y is FSi_others, To calculate x using the inner product j The softmax() function is used to calculate the similarity between points in Y and x. j The matching probability with each point of Y, Calculate x j The corresponding average point, It's x j The target point for adjustment.
[0049] like Figure 2 As shown, after obtaining a series of target points, the registration matrix can be calculated through the registration parameter calculation module. Preferably, the calculation method in this embodiment is singular value decomposition.
[0050] The registration network described in this embodiment needs to be trained. The training process involves calculating the feature matrices of each input point cloud using the registration matrix, comparing the obtained registration matrix with the ground truth, calculating the loss function, and backpropagating it into the network to optimize and obtain the parameters of the registration network. Preferably, the loss function is calculated using the following formula:
[0051]
[0052] Among them, (p,p + ) represents the corresponding pair of points, R p Let p be the coordinates of the point after transformation by the registration network. These are the coordinates of the point under the influence of the truth transformation matrix.
[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.
Claims
1. A method for multi-view point cloud registration based on deep learning, characterized in that, The method comprises the following steps: Step 1: sampling a multi-view point cloud set to be registered to form a point cloud set to be registered; Step 2: for elements in the point cloud set to be registered, a feature extraction method is used to extract and express the features of each point in the element to obtain a feature vector corresponding to each point, and the feature vectors corresponding to different points are combined to form a feature set of each point cloud; Step 3: the feature set of each point cloud is input to calculate a registration matrix through a registration network; Step 4: the point cloud to be registered is registered according to the registration matrix, and each point cloud to be registered is unified to the same coordinate system to generate a complete point cloud model of the object.
2. The multi-view point cloud registration method based on deep learning according to claim 1, wherein, The point cloud set to be registered is sampled by using a key point sampling method; wherein the key point is defined as a local extreme point or a corner point, and the sampling method is realized by using a gradient operator. 3.The multi-view point cloud registration method based on deep learning according to claim 1, wherein, The feature extraction method comprises the following steps: A pre-trained point cloud feature extraction network is used; the initial network parameters are obtained by transfer learning, and then the final network parameters are obtained by training on a task target data set.
4. The multi-view point cloud registration method based on deep learning according to claim 3, characterized in that, The point cloud feature extraction network adopts a PointNet network; a set of extracted features of the point cloud to be registered obtained through the PointNet network is FSi={f j ,j∈N} where f j is an extracted feature of a point in the point cloud to be registered, and N represents the number of features.
5. The multi-view point cloud registration method based on deep learning according to claim 1, characterized in that, The registration network is composed of a Transformer structure and a registration matrix calculation module, wherein the Transformer structure is used to fuse the features of each point cloud to be registered with the features of the remaining point clouds to be registered to introduce global information in the process of point cloud registration, and the registration matrix calculation module is used to calculate the registration matrix.
6. The multi-view point cloud registration method based on deep learning according to claim 5, characterized in that, The Transformer structure takes the extracted feature set FSi of the i-th point cloud to be registered as the input end of the Transformer structure, and takes the extracted feature set FSi_others of the other point clouds as the reference output, calculates the residual φ(FSi, FSi_others), and then FSi corresponds to the fused feature set FSi+ = FSi+ φ(FSi, FSi_others).
7. The multi-view point cloud registration method based on deep learning according to claim 4, characterized in that, The registration matrix calculation module performs the following steps: Pair point generation: for a point x in the extracted feature set FSi j , compute according to the formula ; where Y is the extracted feature set FSi_others of the other point cloud, is the similarity of x j to each point in Y computed by inner product, softmax() is used to compute the matching probability of x j to each point in Y, is the corresponding average point of x j , is the adjusted target point of x j ; Registration parameter computation: The adjusted target points The registration matrix is obtained by singular value decomposition.
8. The multi-view point cloud registration method based on deep learning according to claim 1 or 5, characterized in that, The registration network is obtained by training; the point cloud feature set input by the registration matrix calculation module is compared with the true value to calculate the loss function and back-propagate it to the registration network to optimize and obtain the parameters of the registration network. 9.A multi-view point cloud registration system based on deep learning, characterized in that, It comprises: A point cloud to be registered acquisition unit is configured to sample a multi-view point cloud set to be registered to form a point cloud set to be registered; A feature extraction unit is configured to, for elements in the point cloud set to be registered, use a feature extraction method to extract and express the features of each point in the element to obtain a feature vector corresponding to each point, and combine the feature vectors corresponding to different points to form a feature set of each point cloud; A registration network, which takes the feature set of each point cloud as input, calculates a registration matrix through the registration network; A multi-view point cloud registration unit registers the point cloud to be registered according to the registration matrix, and unifies each point cloud to be registered to the same coordinate system to generate a complete point cloud model of the object.
10. The multi-view point cloud registration system based on deep learning of claim 9, wherein, The registration network comprises: The extracted feature set FS i of the i th point cloud to be registered is taken as an input of a Transformer structure, after first multi-head attention, layer normalization processing is performed, and the FS i is summed through a first residual connection to obtain hidden features FH1, after the FH1 passes through a feedforward layer, normalization processing is performed, and the FH1 is summed through a second residual connection to obtain hidden features FH2; The extracted feature set FS i_others of the other point cloud to be registered is combined and spliced through a combination layer to obtain hidden features FH3, after second multi-head attention, layer normalization processing is performed, and the FH3 is summed through a third residual connection to obtain hidden features FH4; after the FH2 and the FH4 are input into third multi-head attention, normalization layer processing is performed, and the FH4 is summed through a fourth residual connection to obtain hidden features FH5; after the FH5 passes through a second feedforward network, layer normalization processing is performed, and the FH5 is summed through a second residual connection to obtain hidden features FH6, and the FH6 is summed with the FS i through a residual connection layer to obtain a fused feature set FS i+.