Point cloud registration method, device, terminal equipment and medium based on deep learning

Through deep learning methods, multi-scale voxelization and feature fusion of point clouds are solved, and the registration failure problem of ICP algorithm in local maximum and abnormal point cloud situations is achieved, and efficient and accurate point cloud registration is achieved.

CN114022525BActive Publication Date: 2024-11-26CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111319507.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-09
Publication Date
2024-11-26
Estimated Expiration
2041-11-09

AI Technical Summary

Technical Problem

Existing point cloud registration methods such as ICP algorithms are prone to fall into local maximum values ​​and fail when overlapping parts between point clouds are too small or have abnormal points.

Method used

Using a deep learning-based method, multi-scale voxelization segmentation and deep learning feature extraction of point clouds, multi-scale features are fused to obtain overlapping matching features, and the optimal solutions of rotation matrix and translation matrix are calculated for point cloud registration. .

Benefits of technology

The local maximum trap is effectively avoided, and the accuracy and efficiency of point cloud registration are improved, especially in the case of uneven density and noisy point cloud data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114022525B_ABST
    Figure CN114022525B_ABST
Patent Text Reader

Abstract

The present disclosure provides a point cloud registration method, apparatus, terminal device and computer-readable storage medium based on deep learning, wherein the method includes: performing multi-scale voxel segmentation on each group of point clouds respectively to obtain multi-scale voxelized point cloud blocks of each group of point clouds; performing deep learning on the multi-scale voxelized point cloud blocks of each group of point clouds respectively to obtain a number of point cloud features of each group of point clouds; fusing a number of point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds; obtaining overlapping matching features between each group of point clouds based on the fusion features of each group of point clouds; and performing point cloud registration on each group of point clouds based on the overlapping matching features. The present disclosure can quickly process point cloud data with uneven density, avoid falling into a local maximum, and can accurately find the overlapping features of point clouds, which can effectively solve the problems of registration failure caused by too small overlapping parts between point clouds or abnormal points, and improve the efficiency of point cloud registration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a point cloud registration method based on deep learning, a point cloud registration device based on deep learning, a terminal device, and a computer-readable storage medium. Background Art

[0002] 3D point cloud is a disordered representation of the geometric structure of the surface of a 3D object. It can be acquired through 3D vision technology such as 3D depth camera. 3D point cloud can be used in 3D object reconstruction, reverse engineering design, unmanned driving, robotics and other fields. Due to the limitations of 3D vision acquisition equipment and measurement environment, the acquisition of complete point cloud data of 3D objects often requires multiple measurements at different angles and splicing of multiple groups of partial point clouds to obtain complete point cloud data of 3D objects. This task of splicing and integrating partially overlapping point cloud data is called point cloud registration.

[0003] Point cloud registration is an important research direction in computer vision. Iterative Closest Point (ICP) is widely used because of its simplicity and low complexity. It can first construct local geometric features based on point cloud data, and then relocate point cloud data based on local geometric features. However, it is easy to cause point cloud data to fall into local maximum, and when the overlap between two point clouds is too small or there are many noise / abnormal points, it will lead to registration failure. Summary of the invention

[0004] The present disclosure provides a point cloud registration method, apparatus, terminal device and computer-readable storage medium based on deep learning, so as to at least solve the problems that the point cloud registration using the ICP algorithm is prone to fall into the local maximum and the registration fails when the overlap between the point clouds is too small or there are abnormal points.

[0005] According to one aspect of the present disclosure, a point cloud registration method based on deep learning is provided, comprising:

[0006] Perform multi-scale voxel segmentation on each group of point clouds respectively to obtain multi-scale voxel point cloud blocks of each group of point clouds;

[0007] Perform deep learning on the multi-scale voxelized point cloud blocks of each group of point clouds to obtain several point cloud features of each group of point clouds;

[0008] Fusing several point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds;

[0009] Acquire overlapping matching features between each group of point clouds based on fusion features of each group of point clouds; and perform point cloud registration on each group of point clouds based on the overlapping matching features.

[0010] In one embodiment, the deep learning is performed on the multi-scale voxelized point cloud blocks of each group of point clouds to obtain a number of point cloud features of each group of point clouds, including:

[0011] constructing a multi-scale neural network, the multi-scale neural network comprising neural network layers of several different scales; and,

[0012] The multi-scale voxelized point cloud blocks of each group of point clouds are synchronously input into the multi-scale neural network to obtain a number of point cloud features of each group of point clouds.

[0013] In one embodiment, the multiscale neural network includes a first multiscale sub-neural network and a second multiscale sub-neural network, and the multiscale voxelized point cloud blocks of each group of point clouds are synchronously input into the multiscale neural network, including:

[0014] synchronously inputting the multi-scale voxelized point cloud blocks of each group of point clouds into the first multi-scale sub-neural network for down-sampling, and obtaining a number of initial feature extraction results of each group of point clouds; and

[0015] The initial feature extraction results of each group of point clouds are synchronously input into the second multi-scale sub-neural network for upsampling to obtain a number of point cloud features of each group of point clouds.

[0016] In one implementation, several point cloud features of each group of point clouds are fused respectively to obtain fused features of each group of point clouds, including:

[0017] The multi-layer perceptron MLP is used as an encoder to fuse several point cloud features of each group of point clouds respectively to obtain the fused features of each group of point clouds.

[0018] In one implementation, each group of point clouds includes an X group of point clouds and a Y group of point clouds, and a multi-layer perceptron MLP is used as an encoder to fuse several point cloud features of each group of point clouds in different scale neural network layers, and the following formula is used to obtain:

[0019]

[0020]

[0021] In the formula, F X represents the fusion features of X groups of point clouds, F Y represents the fusion features of Y groups of point clouds, MLP represents the operator for fusing each group of point clouds through MLP, represents the output full connection, D represents the total number of multi-scale voxel segmentation, φ d (X, f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks of X group point clouds after multi-scale voxel segmentation, φd (Y,f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks segmented by multi-scale voxelization of Y group point clouds.

[0022] In one implementation, performing point cloud registration on each group of point clouds based on the overlapping matching features includes:

[0023] Calculating the optimal solutions of the rotation matrix and the translation matrix between each group of point clouds based on the overlapping matching features; and performing point cloud registration on each group of point clouds based on the optimal solutions.

[0024] In one implementation, each group of point clouds includes an X group of point clouds and a Y group of point clouds, and the optimal solution of the rotation matrix and the translation matrix between each group of point clouds is calculated based on the overlapping matching feature, which is obtained according to the following formula:

[0025]

[0026] Where R', T' represents the optimal solution of the rotation matrix and translation matrix between the X group point cloud and the Y group point cloud, M represents the overlapping matching features between the X group point cloud and the Y group point cloud, and x i ,y j They are the points belonging to the overlapping matching features of the X group point cloud and the Y group point cloud, R represents the rotation matrix, and T represents the translation matrix.

[0027] According to another aspect of the present disclosure, a point cloud registration device based on deep learning is provided, comprising:

[0028] A segmentation module is configured to perform multi-scale voxel segmentation on each group of point clouds respectively to obtain multi-scale voxelized point cloud blocks of each group of point clouds;

[0029] A training module is configured to perform deep learning on multi-scale voxelized point cloud blocks of each group of point clouds respectively to obtain a number of point cloud features of each group of point clouds;

[0030] A fusion module, which is configured to fuse several point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds;

[0031] an acquisition module, which is configured to acquire overlapping matching features between each group of point clouds based on fusion features of each group of point clouds; and

[0032] A registration module is configured to perform point cloud registration on each group of point clouds based on the overlapping matching features.

[0033] According to another aspect of the present disclosure, a terminal device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the deep learning-based point cloud registration method.

[0034] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the deep learning-based point cloud registration method.

[0035] According to the deep learning-based point cloud registration method provided by the present invention, each group of point clouds is divided into multi-scale voxelized point cloud blocks, and deep learning is performed on the multi-scale voxelized point cloud blocks of each group of point clouds to obtain a number of point cloud features of each group of point clouds, and then the several point cloud features of each group of point clouds are fused to obtain the fusion features of each group of point clouds, and then the overlapping matching features are obtained based on the fusion features to achieve point cloud registration between the groups of point clouds. Point cloud data with uneven density can be processed quickly to avoid falling into local maximum values, and overlapping features of point clouds can be accurately found. The problems of registration failure caused by too small overlapping parts between point clouds or abnormal points can be effectively solved, thereby improving the efficiency of point cloud registration.

[0036] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0038] Figure 1 A schematic diagram of a process of a point cloud registration method based on deep learning provided in an embodiment of the present disclosure;

[0039] Figure 2 for Figure 1 Schematic diagram of the process of step S12;

[0040] Figure 3 A schematic diagram of the process of performing semantic segmentation of point clouds in related technologies;

[0041] Figure 4 for Figure 1 Schematic diagram of the process of step S15;

[0042] Figure 5 A schematic diagram of a process of a point cloud registration method based on deep learning provided in another embodiment of the present disclosure;

[0043] Figure 6A schematic diagram of a scene in which multi-scale voxelized point cloud blocks of X groups of point clouds are synchronously input into a multi-scale neural network in another embodiment of the present disclosure;

[0044] Figure 7 A schematic diagram of the structure of a point cloud registration device based on deep learning provided in an embodiment of the present disclosure;

[0045] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] To make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0048] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.

[0049] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0050] The overall idea of ​​the ICP algorithm in the related technology is to assume that there are two groups of point clouds P and G, and the two have similar local geometric features. It is necessary to find a rotation and translation matrix to minimize the error between P and G after transformation, that is, a rotation and translation matrix R is needed. For each rotation and translation transformation, the distance from all sampling points of P to the corresponding nearest point of G is calculated, and the least squares error is calculated using the least squares method to see if it is within the set range. If the least squares error is less than the set value or the number of iterations reaches the upper limit, the calculation ends, otherwise the iteration continues until the minimum value is reached. The above process is prone to fall into a local maximum, and when the overlapping part between the two point clouds is too small or there are many noise / abnormal points, it will cause the registration to fail.

[0051] To solve the above problems, the disclosed embodiment proposes a 3D point cloud registration method based on deep learning. First, the 3D point cloud is segmented into multi-scale voxels, and feature extraction is performed based on self-supervised deep learning, so as to quickly process 3D point cloud data with uneven density and find the correct overlapping matching relationship between multiple groups of point clouds. After that, the optimal solution of the rotation and translation transformation matrix is ​​obtained through the transformation estimator, and finally efficient point cloud registration is completed, so as to effectively solve the problems in the related technology that the ICP algorithm is prone to fall into the local maximum and easily lead to registration failure.

[0052] Please refer to Figure 1 , Figure 1 A flowchart of a point cloud registration method based on deep learning provided for the implementation of the present disclosure includes steps S11-S15.

[0053] In step S11, multi-scale voxelized segmentation is performed on each group of point clouds to obtain multi-scale voxelized point cloud blocks of each group of point clouds.

[0054] In one implementation, if point cloud registration is required for two point clouds X and Y, X∈P Nx×3 , Y∈P Ny×3 , where P is the point cloud, which refers to an entire point cloud. Nx and Ny represent the scale width of the X and Y point cloud sets on a certain axis in the three-dimensional coordinate system, that is, the number of divided voxel blocks. The larger N is, the smaller the divided voxel blocks are. Multi-scale voxelization is performed on each group of point clouds by multiplying by 2 at each scale of xyz. In some embodiments, multi-scale voxelization can also be performed by multiplying by other values. Suppose that the two groups of point clouds are respectively subjected to multi-scale voxelization segmentation D times, and the two groups of point clouds are divided into A, B, ..., H voxelized point cloud blocks of different scales. For example, by using a point cloud with 3 scale voxels (i.e., multi-scale voxelization segmentation 3 times), the resolution size of the initial voxel can be expanded to 64 times, i.e., 1*1*1=1, 2*2*2=8, 4*4*4=64. In the actual algorithm, the network can dynamically set the initial voxel size and the number of voxel grid sub-sampling times (multi-scale voxelization segmentation times) according to the volume of the point cloud. It can be understood that the initial voxel size is to voxelize the unsegmented original point cloud for the first time, and then perform multi-scale division according to the strategy of multiplying the scale by 2.

[0055] It can be understood that multi-scale segmentation refers to the process of generating meaningful image polygon objects with minimal heterogeneity and maximum homogeneity at any scale under the premise of minimal image information loss. It is a means of image abstraction (compression), that is, retaining the information of high-resolution pixels on low-resolution objects, and different types of land objects can be reflected on objects of corresponding scales. This embodiment uses multi-scale segmentation to perform multi-scale voxel segmentation on multiple groups of point clouds to obtain voxelized three-dimensional models at different scales of multiple groups of point clouds, that is, multi-scale voxelized point cloud blocks A, B, ..., H.

[0056] In step S12, deep learning is performed on the multi-scale voxelized point cloud blocks of each group of point clouds to obtain a number of point cloud features of each group of point clouds.

[0057] Specifically, this embodiment obtains point cloud feature data of each group of point clouds at different scales by inputting the multi-scale voxelized point cloud blocks of each group of point clouds into the neural network layer for deep learning, so as to quickly process 3D point cloud data with uneven density.

[0058] Optionally, by dividing the neural network into neural network layers of different scales, multi-scale voxel blocks are simultaneously fed into the neural network to capture intermediate key feature information in different dimensions in the point cloud, which can bring more global context, which is more effective for point cloud registration. The multi-scale voxelized point cloud blocks of each group of point clouds are deep-learned to obtain a number of point cloud features of each group of point clouds (step S12), such as Figure 2 As shown, the process includes the following steps S121 and S122.

[0059] Since 3D point cloud collection is prone to noise, outliers, incomplete point clouds and other problems, point cloud registration is very difficult. At present, with the continuous maturity of deep learning, point cloud registration problems can be solved by learning point cloud features and point cloud transformation calculation methods on a large number of point cloud data sets. For example, in related technologies, such as Figure 3 As shown, it is used for semantic segmentation of point cloud, and classifies each point by simultaneously inputting several adjacent voxels of different scales, and labels each point. Although it also uses voxels of different scales for semantic segmentation of point cloud, the difference is that it directly classifies each point in the point cloud after segmentation, while this embodiment subsamples the entire 3D point cloud voxel blocks of different voxel scales, and inputs the voxelized point cloud blocks of different scales into a multi-scale neural network for learning, which greatly improves the feature learning performance based on supervised learning.

[0060] Step S121: construct a multi-scale neural network, wherein the multi-scale neural network includes neural network layers of several different scales.

[0061] Step S122: synchronously input the multi-scale voxelized point cloud blocks of each group of point clouds into the multi-scale neural network to obtain a plurality of point cloud features of each group of point clouds.

[0062] For example, the neural network is divided into multiple neural networks of different scales: {3D Conv 3×3×3,1,32} (convolution kernel size 3, step size 1, filter 32), {3D Conv 3×3×3,2,64}, {3D Conv 3×3×3,2,128}, and {3D Conv 3×3×3,2,256}.

[0063] In step S13, several point cloud features of each group of point clouds are fused respectively to obtain fused features of each group of point clouds.

[0064] Specifically, by fusing the features of point clouds at different scales, fusion features are obtained. It can be understood that fusing features of different scales is an important means to improve segmentation performance. For example, the point cloud features obtained by inputting the point cloud into the neural network have higher resolution in the low-level features and contain more position and detail information, but due to fewer convolutions, their semantics are lower and there is more noise. High-level features have stronger semantic information, but their resolution is very low and their perception of details is poor. By efficiently fusing the two, taking their strengths and discarding their weaknesses, is the key to improving the segmentation model.

[0065] In step S14, overlapping matching features between each group of point clouds are obtained based on the fusion features of each group of point clouds.

[0066] In this embodiment, after obtaining the fusion features of each group of point clouds (for example, two groups of point clouds X and Y, whose fusion features are F X and F Y ), based on the fusion features, the overlapping matching relationship between point cloud X and point cloud Y is calculated through the metric learning of the Triplet loss function, and then the overlapping matching feature M is found. Specifically, the Triplet loss function is obtained according to the following formula:

[0067]

[0068] The above formula is explained as follows: L(F, F + , F - ) represents the loss of the overlapping matching relationship between the point pairs in point cloud X and point cloud Y. The smaller the loss, the greater the matching relationship, indicating that the two are overlapping matching features. F represents the overlapping matching feature between any two point pairs. F + Represents the features of positive matching point pairs, F - Denotes the negative matching point pair feature, D(F, F + ) represents the distance between the overlapping matching feature and the positive matching point pair feature, D(F, F- ) represents the distance between the overlapping matching feature and the negative matching point pair feature, and m represents the threshold parameter. Specifically, let (X i ,Y j ) is a given positive matching point pair, that is, if there is a feature overlap between the two points, their fusion output feature F should be Xi and F Yj The distance between them is as close to 0 as possible, but to prevent overfitting, it is only necessary to control the distance between them to be less than a certain threshold, that is, D(F Xi ,F Yj )<m p Similarly, let (X i ,Y j ) is a randomly selected negative matching point pair, that is, there is no feature overlap between the two points, and their fusion output feature F should be Xi and F Yj The distance between them is as close to infinity as possible, but to prevent overfitting, it is only necessary to control the distance between them to be greater than a certain set threshold, that is, D(F Xi ,F Yj )>m n . Set m to be m p and m n Therefore, the above formula is to find the correct overlapping matching feature F so that it is consistent with the feature F of the positive matching point pair. + The closer the better, and with the negative matching point feature F - The farther the better, the network is trained through a given model and loss function to achieve the goal of reasoning about the point cloud set and finding the correct overlapping matching point cloud set M.

[0069] It should be noted that in some embodiments, other loss functions may also be used to obtain overlapping matching features between groups of point clouds, and the above examples are not limitations of the present disclosure. By introducing the Triplet loss function, this embodiment can continuously optimize the model parameters to minimize the loss function L value, and then achieve the purpose of self-supervised optimization of neural network parameters, so that the obtained overlapping matching features M are more accurate. It can be understood that the overlapping matching feature M is a collection of feature overlapping points between each group of point clouds.

[0070] In step S15, point cloud registration is performed on each group of point clouds based on the overlapping matching features.

[0071] Optionally, the optimal solution of the rotation matrix and translation matrix between each group of point clouds can be calculated for the overlapping matching features to achieve point cloud registration, such as Figure 4 As shown, step S15 is specifically as follows:

[0072] Step S151, calculating the optimal solution of the rotation matrix and the translation matrix between each group of point clouds based on the overlapping matching features; and step S152, performing point cloud registration on each group of point clouds based on the optimal solution.

[0073] In one implementation, the optimal solution of the rotation matrix and translation matrix between each group of point clouds can be calculated for the overlapping matching features, and the difference between each point in each group of point clouds after rotation and translation is minimized based on the optimal solution. For example, for the X group of point clouds and the Y group of point clouds, the optimal solution of the rotation matrix R and the translation matrix T is obtained by using transformation estimators such as FGR, AO, ICL and other algorithms, as shown in the following formula, arg min refers to the value of the variable when the following formula reaches the minimum value. That is, the optimal parameter solution of R and T is obtained so that the difference between each point in the X point cloud and each point in the Y point cloud after rotation and translation is minimized, so as to achieve the final efficient point cloud registration.

[0074] Specifically, the optimal solution for calculating the rotation matrix R and the translation matrix T in step S151 is obtained according to the following formula:

[0075]

[0076] Where R', T' represents the optimal solution of the rotation matrix and translation matrix between the X group point cloud and the Y group point cloud, M represents the overlapping matching features between the X group point cloud and the Y group point cloud, and x i represents the i-th point in the X group of point clouds that belongs to the overlapping matching feature, y j It represents the jth point in the Y group of point clouds that belongs to the overlapping matching feature, R represents the rotation matrix, and T represents the translation matrix.

[0077] Please refer to Figure 5 , Figure 5 A flow chart of a point cloud registration method based on deep learning is provided for another embodiment of the present disclosure, wherein the neural network layer is also divided into neural network layers of different scales, wherein the multi-scale neural network includes a first multi-scale sub-neural network and a second multi-scale sub-neural network, and voxelized point cloud blocks of different scales are synchronously input into the first multi-scale sub-neural network for down-sampling and then input into the second multi-scale sub-neural network for up-sampling to obtain more accurate point cloud features. Specifically, the multi-scale voxelized point cloud blocks of each group of point clouds are synchronously input into the multi-scale neural network (step S122), including the following steps S1221 and S1222.

[0078] S1221, synchronously inputting the multi-scale voxelized point cloud blocks of each group of point clouds into the first multi-scale sub-neural network for down-sampling, to obtain a number of initial feature extraction results of each group of point clouds;

[0079] S1222. The initial feature extraction results of each group of point clouds are synchronously input into the second multi-scale sub-neural network for upsampling to obtain a plurality of point cloud features of each group of point clouds.

[0080] Combination Figure 6 As shown in the figure, taking X groups of point clouds as an example, the divided multi-scale voxelized point cloud blocks A, B, ..., H are synchronously input into the multi-scale neural network for feature extraction. First, the point cloud voxel blocks are downsampled in series and input into the neural network layers of {3DConv 3×3×3,1,32} (convolution kernel size 3, step size 1, filter 32), {3D Conv 3×3×3,2,64}, {3DConv 3×3×3,2,128}, and {3D Conv 3×3×3,2,256} in sequence to obtain the initial feature extraction results. The use of a series of downsampling methods can capture the intermediate key feature information in different dimensions in the point cloud and bring more global context.

[0081] Then, for the feature extraction results of the voxelized point cloud blocks of each scale obtained by downsampling, a series of upsampling operations are performed and input into the neural network layers of {3D Conv 3×3×3,2,128}, {3D Conv 3×3×3,2,64}, and {3D Conv 3×3×3,1,32} in sequence. Upsampling can accurately locate the feature information globally, and obtain the point cloud features A, B, ..., H corresponding to the voxelized point cloud blocks A, B, ..., H of each scale of the X group of point clouds, and then input them into the fully connected layer for feature fusion.

[0082] Furthermore, after obtaining a number of point cloud features of each group of point clouds, this embodiment directly inputs them into the hidden layer, and uses the fully connected layer (Fully Connected Layer, FC) in the hidden layer, that is, the multi-layer perceptron (MLP), to perform feature fusion, and fuses a number of point cloud features of each group of point clouds respectively to obtain the fusion features of each group of point clouds. Specifically, the following steps are performed:

[0083] The multi-layer perceptron MLP is used as an encoder to fuse several point cloud features of each group of point clouds respectively to obtain the fused features of each group of point clouds.

[0084] Furthermore, each group of point clouds includes an X group of point clouds and a Y group of point clouds, and a multi-layer perceptron MLP is used as an encoder to fuse several point cloud features of each group of point clouds in different scale neural network layers, and the following formula is used to obtain:

[0085]

[0086]

[0087] In the formula, F X represents the fusion features of X groups of point clouds, F Y represents the fusion features of Y groups of point clouds, MLP represents the operator for fusing each group of point clouds through MLP, represents the output full connection, D represents the total number of multi-scale voxel segmentation, φ d (X, f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks of X group point clouds after multi-scale voxel segmentation, φ d (Y,f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks segmented by multi-scale voxelization of Y group point clouds.

[0088] Based on the same technical concept, the embodiment of the present disclosure also provides a point cloud registration device based on deep learning, such as Figure 7 As shown, the device includes a segmentation module 71, a training module 72, a fusion module 73, an acquisition module 74 and a registration module 75, wherein:

[0089] The segmentation module 71 is configured to perform multi-scale voxel segmentation on each group of point clouds to obtain multi-scale voxel point cloud blocks of each group of point clouds;

[0090] The training module 72 is configured to perform deep learning on the multi-scale voxelized point cloud blocks of each group of point clouds respectively to obtain a number of point cloud features of each group of point clouds;

[0091] The fusion module 73 is configured to fuse several point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds;

[0092] The acquisition module 74 is configured to acquire overlapping matching features between each group of point clouds based on the fusion features of each group of point clouds; and

[0093] The registration module 75 is configured to perform point cloud registration on each group of point clouds based on the overlapping matching features.

[0094] Furthermore, the training module 72 includes:

[0095] a construction unit configured to construct a multi-scale neural network, wherein the multi-scale neural network includes neural network layers of several different scales; and

[0096] An input unit is configured to synchronously input the multi-scale voxelized point cloud blocks of each group of point clouds into the multi-scale neural network to obtain a plurality of point cloud features of each group of point clouds.

[0097] Furthermore, the multiscale neural network includes a first multiscale sub-neural network and a second multiscale sub-neural network.

[0098] The input unit is specifically configured to synchronously input the multi-scale voxelized point cloud blocks of each group of point clouds into the first multi-scale sub-neural network for down-sampling to obtain a number of initial feature extraction results of each group of point clouds; and synchronously input the initial feature extraction results of each group of point clouds into the second multi-scale sub-neural network for up-sampling to obtain a number of point cloud features of each group of point clouds.

[0099] Furthermore, the fusion module 73 is specifically configured to use a multi-layer perceptron MLP as an encoder to fuse several point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds.

[0100] Furthermore, each group of point clouds includes an X group of point clouds and a Y group of point clouds. The fusion module 73 uses a multi-layer perceptron MLP as an encoder to fuse several point cloud features of each group of point clouds in different scale neural network layers, and obtains the following formula:

[0101]

[0102]

[0103] In the formula, F X represents the fusion features of X groups of point clouds, F Y represents the fusion features of Y groups of point clouds, MLP represents the operator for fusing each group of point clouds through MLP, represents the output full connection, D represents the total number of multi-scale voxel segmentation, φ d (X, f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks of X group point clouds after multi-scale voxel segmentation, φ d (Y,f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks segmented by multi-scale voxelization of Y group point clouds.

[0104] Furthermore, the registration module 75 includes:

[0105] A calculation unit, which is configured to calculate the optimal solution of the rotation matrix and the translation matrix between each group of point clouds based on the overlapping matching features; and a registration unit, which is configured to perform point cloud registration on each group of point clouds based on the optimal solution.

[0106] Further, each group of point clouds includes an X group of point clouds and a Y group of point clouds, and the calculation unit is obtained according to the following formula:

[0107]

[0108] Where R', T' represents the optimal solution of the rotation matrix and translation matrix between the X group point cloud and the Y group point cloud, M represents the overlapping matching features between the X group point cloud and the Y group point cloud, and x i represents the i-th point in the X group of point clouds that belongs to the overlapping matching feature, y j It represents the jth point in the Y group of point clouds that belongs to the overlapping matching feature, R represents the rotation matrix, and T represents the translation matrix.

[0109] Based on the same technical concept, the embodiment of the present disclosure also provides a terminal device, such as Figure 8 As shown, the terminal device includes a memory 81 and a processor 82, wherein the memory 81 stores a computer program. When the processor 82 runs the computer program stored in the memory 81, the processor 82 executes the deep learning-based point cloud registration method.

[0110] Based on the same technical concept, the embodiment of the present disclosure also provides a corresponding computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the deep learning-based point cloud registration method.

[0111] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A point cloud registration method based on deep learning, characterized in that: include: Perform multi-scale voxel segmentation on each group of point clouds respectively to obtain multi-scale voxel point cloud blocks of each group of point clouds; Perform deep learning on the multi-scale voxelized point cloud blocks of each group of point clouds to obtain several point cloud features of each group of point clouds; Fusing several point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds; Obtain overlapping matching features between each group of point clouds based on the fusion features of each group of point clouds; And, performing point cloud registration on each group of point clouds based on the overlapping matching features.

2. The method according to claim 1, characterized in that: The multi-scale voxelized point cloud blocks of each group of point clouds are subjected to deep learning to obtain several point cloud features of each group of point clouds, including: constructing a multi-scale neural network, the multi-scale neural network comprising neural network layers of several different scales; and, The multi-scale voxelized point cloud blocks of each group of point clouds are synchronously input into the multi-scale neural network to obtain a number of point cloud features of each group of point clouds.

3. The method according to claim 2, characterized in that The multiscale neural network includes a first multiscale sub-neural network and a second multiscale sub-neural network, and the multiscale voxelized point cloud blocks of each group of point clouds are synchronously input into the multiscale neural network, including: synchronously inputting the multi-scale voxelized point cloud blocks of each group of point clouds into the first multi-scale sub-neural network for down-sampling, and obtaining a number of initial feature extraction results of each group of point clouds; and The initial feature extraction results of each group of point clouds are synchronously input into the second multi-scale sub-neural network for upsampling to obtain a number of point cloud features of each group of point clouds.

4. The method according to any one of claims 1 to 3, characterized in that: Several point cloud features of each group of point clouds are fused separately to obtain the fusion features of each group of point clouds, including: The multi-layer perceptron MLP is used as an encoder to fuse several point cloud features of each group of point clouds respectively to obtain the fused features of each group of point clouds.

5. The method according to claim 4, characterized in that Each group of point clouds includes an X group of point clouds and a Y group of point clouds. A multi-layer perceptron MLP is used as an encoder to fuse several point cloud features of each group of point clouds in different scale neural network layers, and the following formula is used to obtain: In the formula, F X represents the fusion features of X groups of point clouds, F Y represents the fusion features of the Y group of point clouds, MLP represents the operator for fusing each group of point clouds through MLP, ⊕ represents the output full connection, D represents the total number of multi-scale voxel segmentation, φ d (X, f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks of X group point clouds after multi-scale voxel segmentation, φ d (Y,f d ) represents the point cloud features obtained by deep learning of d-scale voxel point cloud blocks segmented by multi-scale voxelization of Y group point clouds.

6. The method according to claim 1, characterized in that Performing point cloud registration on each group of point clouds based on the overlapping matching features includes: Calculating the optimal solutions of the rotation matrix and the translation matrix between each group of point clouds based on the overlapping matching features; and performing point cloud registration on each group of point clouds based on the optimal solutions.

7. The method according to claim 6, characterized in that Each group of point clouds includes an X group of point clouds and a Y group of point clouds. The optimal solution of the rotation matrix and the translation matrix between each group of point clouds is calculated based on the overlapping matching features, and is obtained according to the following formula: Where R', T' represents the optimal solution of the rotation matrix and translation matrix between the X group point cloud and the Y group point cloud, M represents the overlapping matching features between the X group point cloud and the Y group point cloud, and x i represents the i-th point in the X group of point clouds that belongs to the overlapping matching feature, y j It represents the jth point in the Y group of point clouds that belongs to the overlapping matching feature, R represents the rotation matrix, and T represents the translation matrix.

8. A point cloud registration device based on deep learning, characterized in that: include: A segmentation module is configured to perform multi-scale voxel segmentation on each group of point clouds respectively to obtain multi-scale voxelized point cloud blocks of each group of point clouds; A training module is configured to perform deep learning on multi-scale voxelized point cloud blocks of each group of point clouds respectively to obtain a number of point cloud features of each group of point clouds; A fusion module, which is configured to fuse several point cloud features of each group of point clouds respectively to obtain fusion features of each group of point clouds; An acquisition module, which is configured to acquire overlapping matching features between each group of point clouds based on fusion features of each group of point clouds; as well as, A registration module is configured to perform point cloud registration on each group of point clouds based on the overlapping matching features.

9. A terminal device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the deep learning-based point cloud registration method described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the processor executes the deep learning-based point cloud registration method described in any one of claims 1 to 6.