Search-based multi-view point cloud registration method and device

By introducing search steps in multi-view point cloud registration, searching related point clouds with global feature descriptors and building pose maps, the noise problem caused by no overlapping point clouds in the prior art is solved, and the robustness and The optimization performance of the pose map enables the recovery of the complete scene.

CN114926510BActive Publication Date: 2024-11-01TSINGHUA UNIVERSITY
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210480425.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-05
Publication Date
2024-11-01
Estimated Expiration
2042-05-05

AI Technical Summary

Technical Problem

The existing multi-view point cloud registration method generates incorrect alignment due to the lack of overlapping point clouds, which increases noise, which is not conducive to the optimization of pose maps and reduces the robustness of multi-view point cloud registration.

Method used

Using a search-based method, multiple out-of-order point clouds are collected from the video data, local feature descriptors are extracted and aggregated into global feature descriptors, and relevant point clouds are searched through Euclidean distance or cosine similarity to construct pose maps. Generate low noise relative transformations to optimize the pose map.

Benefits of technology

It effectively improves the robustness of multi-view point cloud registration, ensures the recovery of complete 3D scenes in multi-view scenes, improves the optimization performance of pose maps, and meets the needs of multi-view point cloud registration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114926510B_ABST
    Figure CN114926510B_ABST
Patent Text Reader

Abstract

The present application discloses a search-based multi-view point cloud registration method and device, wherein the method includes: collecting multiple disordered point clouds from video data, extracting local feature descriptors of each point cloud, and aggregating them into multiple global feature descriptors; searching for at least one related point cloud in the remaining point clouds other than each point cloud based on the Euclidean distance or cosine similarity between global feature descriptors, generating a search set for each point cloud; constructing a pose graph according to the search set of each point cloud, and performing multi-view point cloud registration based on the pose graph to restore a complete 3D scene. Thus, the technical problem of using non-overlapping point clouds for registration in the related art, resulting in the generation of incorrectly aligned point cloud pairs, causing relative transformation to increase noise, being unfavorable for the optimization of the pose graph, and reducing the robustness of multi-view point cloud registration is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a search-based multi-view point cloud registration method and device. Background Art

[0002] In related technologies, due to the limitation of the sensor's viewing angle, a single scan cannot obtain a complete 3D scene. Therefore, it is necessary to align the point clouds scanned by the sensor from multiple viewing angles to generate a complete 3D scene. Point cloud alignment, namely point cloud registration, is a key issue in computer vision and plays an important role in applications such as robotics, autonomous driving, and augmented reality.

[0003] However, the related technology uses non-overlapping point clouds for registration, which results in the generation of incorrectly aligned point cloud pairs, causing relative transformation to increase noise, which is not conducive to the optimization of the pose graph and reduces the robustness of multi-view point cloud registration, which needs to be solved urgently. Summary of the invention

[0004] This application is based on the inventor's understanding and discovery of the following problems:

[0005] Recently, deep learning-based point cloud registration methods, such as FCGF (Fully-Convolutional Geometric Features), Predator, PointDSC, etc., have made great progress. However, these registration methods are mainly paired registration methods, which can only register two point clouds at a time. The two registered point clouds are usually only part of the entire scene. Therefore, the paired registration method cannot complete the task of indoor scene reconstruction. In contrast, multi-view point cloud registration is to restore the complete scene from a series of disordered multi-view point clouds (i.e., scene fragments). Multi-view point cloud registration provides a solution for indoor scene reconstruction. The first step of the existing multi-view point cloud registration method is to estimate the unordered point clouds through paired registration. The relative transformation between the point clouds is required, and then the optimization with global consistency constraints is used to solve the absolute transformation of the multi-view point clouds. The latter step is usually regarded as a pose graph optimization problem. In this problem, the pose graph is generated by relative transformation. Low-noise relative transformation is helpful for pose graph optimization. However, in multi-view point cloud registration, the point clouds are captured from different viewpoints in the scene, which will cause some point clouds to have no overlap with other point clouds. If the non-overlapping point clouds are registered, incorrectly aligned point cloud pairs will be generated, resulting in noisy relative transformations. High-noise relative transformations will be introduced into the subsequent pose graph optimization steps, which will have a negative impact on existing multi-view point cloud registration methods.

[0006] The present application provides a search-based multi-view point cloud registration method and device to solve the problems in the related art of using non-overlapping point clouds for registration, resulting in the generation of incorrectly aligned point cloud pairs, causing relative transformation to increase noise, which is not conducive to the optimization of the pose graph, reduces the robustness of the multi-view point cloud registration, and cannot meet the requirements of multi-view point cloud registration.

[0007] The first aspect of the present application provides a search-based multi-view point cloud registration method, comprising the following steps: collecting multiple disordered point clouds from video data, extracting local feature descriptors of each point cloud, and aggregating them into multiple global feature descriptors; based on the Euclidean distance or cosine similarity between the global feature descriptors, searching for at least one related point cloud in the remaining point clouds other than each point cloud, and generating a search set for each point cloud; constructing a pose graph according to the search set of each point cloud, and performing multi-view point cloud registration based on the pose graph to restore the complete 3D scene.

[0008] Optionally, in one embodiment of the present application, based on the Euclidean distance or cosine similarity between the global feature descriptors, searching for at least one related point cloud in the remaining point clouds except each point cloud to generate a search set for each point cloud, including: taking any point cloud as a query point cloud, and taking point clouds except any point cloud as candidate point clouds; respectively obtaining the global descriptors of the query point cloud and the candidate point cloud, calculating the Euclidean distance or cosine similarity between the global descriptors of the query point cloud and the candidate point cloud, and obtaining a similarity score between the two point clouds; screening point clouds with similarity scores greater than a preset threshold, and determining at least one search point cloud for each point cloud.

[0009] Optionally, in one embodiment of the present application, constructing a pose graph according to the search set of each point cloud includes: based on the search set of each point cloud, aligning the query point cloud and the search point cloud in the search set by pairwise alignment to obtain a relative transformation; and obtaining the pose graph using the relative transformation and the similarity score.

[0010] Optionally, in one embodiment of the present application, the step of obtaining the pose graph using the relative transformation and the similarity score includes: using each point cloud as a vertex of the pose graph, using the relative transformation between two point clouds as an edge, and using the similarity score between the two point clouds as the weight of the edge to generate the pose graph.

[0011] Optionally, in one embodiment of the present application, performing multi-view point cloud registration based on the pose graph includes: solving a global transformation matrix for each point cloud; and performing multi-view point cloud registration based on the global transformation matrix.

[0012] The second aspect of the present application provides a search-based multi-view point cloud registration device, including: an aggregation module, used to collect multiple disordered point clouds from video data, and extract local feature descriptors of each point cloud, and aggregate them into multiple global feature descriptors; a search module, used to search for at least one related point cloud in the remaining point clouds other than each point cloud based on the Euclidean distance or cosine similarity between the global feature descriptors, and generate a search set for each point cloud; a registration module, used to construct a pose graph according to the search set of each point cloud, and perform multi-view point cloud registration based on the pose graph to restore the complete 3D scene.

[0013] Optionally, in one embodiment of the present application, the search module includes: a computing unit, used to take any point cloud as a query point cloud, and take point clouds other than the any point cloud as candidate point clouds, respectively obtain global descriptors of the query point cloud and the candidate point cloud, calculate the Euclidean distance or cosine similarity between the global descriptors of the query point cloud and the candidate point cloud, and obtain a similarity score between the two point clouds; an acquisition unit, used to screen point clouds whose similarity scores are greater than a preset threshold, and determine at least one search point cloud for each point cloud.

[0014] Optionally, in one embodiment of the present application, the registration module is further used to align the query point cloud and the search point cloud in the search set based on the search set of each point cloud by pairwise registration to obtain a relative transformation, and obtain the pose graph using the relative transformation and the similarity score.

[0015] Optionally, in one embodiment of the present application, the registration module is also used to generate the pose graph using each point cloud as a vertex of the pose graph, the relative transformation between the two point clouds as an edge, and the similarity score between the two point clouds as the weight of the edge.

[0016] Optionally, in one embodiment of the present application, the registration module includes: a solving unit, used to solve the global transformation matrix of each point cloud; and a registration unit, used to perform multi-view point cloud registration based on the global transformation matrix.

[0017] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the search-based multi-view point cloud registration method as described in the above embodiment.

[0018] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the search-based multi-view point cloud registration method as described in the above embodiment.

[0019] The embodiment of the present application can search for at least one related point cloud in the remaining point clouds except each point cloud, generate a search set for each point cloud, thereby constructing a pose graph, and performing multi-view point cloud registration based on the pose graph to restore a complete 3D scene, so that a search step is introduced in the multi-view point cloud registration process to significantly remove the noise in the relative transformation, thereby performing multi-view scene point cloud registration more robustly, and then a complete 3D scene can be restored, which is beneficial to the optimization of the pose graph, effectively improving the robustness of the multi-view point cloud registration, and meeting the requirements of the multi-view point cloud registration. As a result, the technical problem of using non-overlapping point clouds for registration in the related art, resulting in the generation of incorrectly aligned point cloud pairs, causing the relative transformation to increase noise, which is not conducive to the optimization of the pose graph and reduces the robustness of the multi-view point cloud registration, is solved.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a multi-view point cloud registration method based on search provided according to an embodiment of the present application;

[0023] Figure 2 A flowchart of a multi-view point cloud registration method based on search provided according to a specific embodiment of the present application;

[0024] Figure 3 is a schematic structural diagram of a multi-view point cloud registration device based on search according to an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] The following describes the search-based multi-view point cloud registration method and device of the embodiment of the present application with reference to the accompanying drawings. In view of the technical problem that the related technology mentioned in the background technology center above uses non-overlapping point clouds for registration, resulting in the generation of incorrectly aligned point cloud pairs, causing relative transformation to increase noise, which is not conducive to the optimization of the pose graph and reduces the robustness of the multi-view point cloud registration, the present application provides a search-based multi-view point cloud registration method, in which the embodiment of the present application can search for at least one related point cloud in the remaining point clouds except each point cloud, generate a search set for each point cloud, thereby constructing a pose graph, and perform multi-view point cloud registration based on the pose graph to restore the complete 3D scene, so that the search step is introduced in the multi-view point cloud registration process to greatly remove the noise in the relative transformation, so as to perform multi-view scene point cloud registration more robustly, and then a complete 3D scene can be restored, which is conducive to the optimization of the pose graph, effectively improves the robustness of the multi-view point cloud registration, and meets the multi-view point cloud registration requirements. As a result, the technical problem in the related art of using non-overlapping point clouds for registration, resulting in the generation of incorrectly aligned point cloud pairs, causing relative transformation to increase noise, being unfavorable for the optimization of the pose graph, and reducing the robustness of the multi-view point cloud registration is solved.

[0028] Specifically, Figure 1 A flowchart of a search-based multi-view point cloud registration method provided in an embodiment of the present application.

[0029] like Figure 1 As shown, the search-based multi-view point cloud registration method includes the following steps:

[0030] In step S101, a plurality of disordered point clouds are collected from video data, and local feature descriptors of each point cloud are extracted and aggregated into a plurality of global feature descriptors.

[0031] It can be understood that in order to restore the complete scene from a series of disordered multi-view point clouds (i.e., scene fragments), the embodiment of the present application first collects multiple disordered point clouds from the video data and extracts the local feature descriptors of each point cloud. There are many methods for extracting the local feature descriptors of each point cloud, such as PointNetVLAD, which is a point cloud search method based on deep learning. It uses PointNet to extract local feature descriptors of point clouds, and then aggregates these local feature descriptors into global feature descriptors using NetVLAD, thereby providing a solution for indoor scene reconstruction when aligning multi-view point clouds.

[0032] As a possible implementation method, assuming that the input of the embodiment of the present application is n disordered point clouds, the embodiment of the present application will be described in detail below.

[0033] For example, an embodiment of the present application randomly samples n depth maps separated by k frames from a video captured by an RGB-D camera and shuffles the order of the n depth maps, and then uses the internal parameters of the RGB-D camera to convert the depth image into a point cloud, thereby obtaining a point cloud set C containing n random point clouds, where k is less than or equal to 15 and n is greater than or equal to 10. It is worth noting that each point cloud is part of the scene, but each point cloud does not necessarily overlap with each other.

[0034] PointNetVLAD is a point cloud search method based on deep learning. It uses PointNet to extract local feature descriptors of point clouds, and then aggregates these local feature descriptors into global feature descriptors using NetVLAD. It then uses a fully connected layer to reduce the dimension of the global descriptor to obtain the final output global descriptor, and searches by calculating the Euclidean distance or cosine similarity between global descriptors.

[0035] In step S102, based on the Euclidean distance or cosine similarity between global feature descriptors, at least one related point cloud in the remaining point clouds except each point cloud is searched to generate a search set for each point cloud.

[0036] It can be understood that the embodiments of the present application can calculate the Euclidean distance or cosine similarity between global descriptors to search for at least one related point cloud in the remaining point clouds except each point cloud, and search each point cloud to obtain subsets of the point clouds, each subset being composed of the query point cloud and other related point clouds.

[0037] Furthermore, in one embodiment of the present application, based on the Euclidean distance or cosine similarity between global feature descriptors, at least one related point cloud in the remaining point clouds other than each point cloud is searched to generate a search set for each point cloud, including: taking any point cloud as a query point cloud, and taking point clouds other than any point cloud as candidate point clouds; respectively obtaining the global descriptors of the query point cloud and the candidate point cloud, calculating the Euclidean distance or cosine similarity between the global descriptors of the query point cloud and the candidate point cloud, and obtaining a similarity score between the two point clouds; screening point clouds with a similarity score greater than a preset threshold, and determining at least one search point cloud for each point cloud.

[0038] For example, the embodiment of the present application can calculate the similarity score through point cloud search and determine at least one search point cloud for each point cloud. For example, the embodiment of the present application uses a search module to search for point clouds related to it in the remaining n-1 point clouds for each point cloud. Specifically, the embodiment of the present application uses a point cloud in set C as a query point cloud and the remaining n-1 point clouds as candidate point clouds. Then, the query point cloud and the candidate point cloud are respectively sent to PointNetVLAD, and the global descriptors of the query point cloud and the candidate point cloud are output, thereby calculating the cosine similarity between the global descriptors as the similarity score of the two point clouds, and selecting the point cloud with a similarity score greater than a preset threshold from the candidate point cloud. The selected point cloud is the searched point cloud. The embodiment of the present application performs a search operation on each point cloud in C, so that n subsets of C can be obtained, and each subset Ci consists of a query point cloud and ki point clouds related to it.

[0039] It should be noted that the specific value of the threshold is set by those skilled in the art according to actual conditions and is not specifically limited here.

[0040] In step S103, a pose graph is constructed according to the search set of each point cloud, and multi-view point cloud registration is performed based on the pose graph to restore the complete 3D scene.

[0041] Specifically, the embodiment of the present application introduces a search module into the multi-view point cloud registration process, and based on the Euclidean distance or cosine similarity between global feature descriptors, searches for at least one related point cloud in the remaining point clouds other than each point cloud, generates a search set for each point cloud, thereby obtaining a low-noise relative transformation, constructs a pose graph according to the search set of each point cloud, and performs multi-view point cloud registration based on the pose graph, thereby restoring a complete 3D scene.

[0042] Among them, in one embodiment of the present application, a pose graph is constructed according to the search set of each point cloud, including: based on the search set of each point cloud, using pairwise alignment to align the query point cloud and the search point cloud in the search set to obtain a relative transformation; and using the relative transformation and the similarity score to obtain the pose graph.

[0043] It can be understood that, in the embodiment of the present application, each search set obtained through the search uses pairwise registration to align the query point cloud and the search point cloud in the search set to obtain a relative transformation, thereby constructing a pose graph using the relative transformation and the similarity score. Among them, the search module is introduced into the multi-view point cloud registration process, which can avoid generating incorrectly aligned point cloud pairs when aligning non-overlapping point clouds, thereby reducing the noise in the relative transformation and effectively improving the optimization performance of the pose graph.

[0044] In the actual implementation process, the embodiment of the present application uses pairwise registration to align the query point cloud and the related point cloud in each subset of C obtained by the search to obtain a relative transformation, thereby constructing a pose graph using the generated relative transformation and similarity score. For example, if there is a relative transformation between the two nodes of an edge, the edge is generated in the pose graph, and the edge weight is initialized by the above-mentioned similarity score.

[0045] Optionally, in one embodiment of the present application, a pose graph is obtained using relative transformation and similarity scores, including: using each point cloud as a vertex of the pose graph, using the relative transformation between two point clouds as an edge, and using the similarity score between the two point clouds as the weight of the edge to generate a pose graph.

[0046] It can be understood that the point cloud represents the vertices of the pose graph, the edge of the pose graph represents the relative transformation between two vertices (i.e., two point clouds), and the similarity score between the two point clouds (calculated by the point cloud search module) is used as the weight of the edge of the pose graph, thereby constructing a pose graph. For example, if there is a relative transformation between the two nodes of an edge, this edge is generated in the pose graph, and the edge weight is initialized by the above-mentioned similarity score.

[0047] Furthermore, in one embodiment of the present application, multi-view point cloud registration is performed based on a pose graph, including: solving a global transformation matrix for each point cloud; and performing multi-view point cloud registration based on the global transformation matrix.

[0048] It can be understood that after obtaining the pose graph, the embodiment of the present application solves the global transformation matrix for each point cloud through a closed-form solution, and then performs multi-view point cloud registration. Among them, the global transformation can include a global rotation transformation and a global translation transformation. The embodiment of the present application can calculate the global rotation transformation and the global translation transformation respectively.

[0049] For example, global rotation transformation: construct a 3n×3n Laplace matrix L, the content of which is the relative rotation transformation of two point clouds. That is:

[0050]

[0051] Where I represents the identity matrix, w ij is the similarity score between point cloud i and point cloud j, and N(i) represents the set of neighbor vertices of vertex i.

[0052] Select the three smallest eigenvalues ​​and calculate the corresponding three eigenvectors. These three eigenvectors are pieced together to form a 3n×3 matrix, which can be regarded as the concatenation of n 3×3 matrices. Furthermore, the embodiment of the present application can perform SVD decomposition on each 3×3 matrix to obtain the global rotation transformation of each point cloud.

[0053] Global translation transformation: introduce a column vector b=b with a dimension of 3n×1 1 T ,…,b n T ) T ,in:

[0054]

[0055] in, is the rotation transformation in the relative transformation calculated in step S4, t ij is the translation transformation in the relative transformation. In this embodiment, the least square method can be used to fit Lt=b to obtain the translation transformation

[0056] At this point, the global rotation transformation and global translation transformation of each point cloud, that is, the global transformation, are obtained. The embodiment of the present application can perform a global transformation on each point cloud separately to obtain the complete scene after the final multi-view point cloud registration.

[0057] like Figure 2 As shown, the working principle of the method of the embodiment of the present application is described in detail with a specific embodiment.

[0058] In step S201 , n unordered point clouds are input.

[0059] That is, multiple out-of-order point clouds are collected from video data.

[0060] In step S202, the point cloud is searched and the similarity score is calculated.

[0061] That is, at least one related point cloud is searched in the remaining point clouds except each point cloud to generate a search set for each point cloud.

[0062] In step S203, pairwise registration is performed. For each subset of C obtained through the search, the embodiment of the present application may use pairwise registration to align the query point cloud and the related point cloud in the subset to obtain a relative transformation.

[0063] In step S204, a pose graph is constructed. In the embodiment of the present application, a pose graph can be obtained by using relative transformation and similarity score, that is, each point cloud is used as a vertex of the pose graph, and the relative transformation between two point clouds is used as an edge, and the similarity score between the two point clouds is used as the weight of the edge to generate a pose graph.

[0064] In step S205, the global transformation of each point cloud is calculated. The global transformation can be composed of a global rotation transformation and a global translation transformation. In the embodiment of the present application, the global rotation transformation and the global translation transformation of each point cloud, that is, the global transformation, can be obtained by respectively calculating.

[0065] In step S206, the complete scene is outputted. In the embodiment of the present application, each point cloud can be globally transformed to obtain the complete scene after the final multi-view point cloud registration.

[0066] According to the search-based multi-view point cloud registration method proposed in the embodiment of the present application, at least one related point cloud in the remaining point clouds except each point cloud can be searched to generate a search set for each point cloud, thereby constructing a pose graph, and performing multi-view point cloud registration based on the pose graph to restore the complete 3D scene. In this way, a search step is introduced into the multi-view point cloud registration process to significantly remove the noise in the relative transformation, thereby performing multi-view scene point cloud registration more robustly, and further restoring a complete 3D scene, which is beneficial to the optimization of the pose graph, effectively improving the robustness of the multi-view point cloud registration, and meeting the requirements of multi-view point cloud registration.

[0067] Next, a search-based multi-view point cloud registration device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0068] Figure 3 It is a block diagram of a search-based multi-view point cloud registration device according to an embodiment of the present application.

[0069] like Figure 3 As shown, the search-based multi-view point cloud registration device 10 includes: an aggregation module 100 , a search module 200 and a registration module 300 .

[0070] Specifically, the aggregation module 100 is used to collect multiple disordered point clouds from the video data, extract the local feature descriptors of each point cloud, and aggregate them into multiple global feature descriptors.

[0071] The search module 200 is used to search for at least one related point cloud in the remaining point clouds except each point cloud based on the Euclidean distance or cosine similarity between the global feature descriptors, and generate a search set for each point cloud.

[0072] The registration module 300 is used to construct a pose graph according to the search set of each point cloud, and perform multi-view point cloud registration based on the pose graph to restore the complete 3D scene.

[0073] Optionally, in one embodiment of the present application, the search module 200 includes: a calculation unit and an acquisition unit.

[0074] Among them, the calculation unit is used to take any point cloud as a query point cloud, and take point clouds other than any point cloud as candidate point clouds, respectively obtain global descriptors of the query point cloud and the candidate point cloud, calculate the Euclidean distance or cosine similarity between the global descriptors of the query point cloud and the candidate point cloud, and obtain the similarity score between the two point clouds.

[0075] The acquisition unit is used to screen point clouds with similarity scores greater than a preset threshold and determine at least one search point cloud for each point cloud.

[0076] Optionally, in one embodiment of the present application, the registration module 300 is further used to align the query point cloud and the search point cloud in the search set based on the search set of each point cloud by pairwise registration to obtain a relative transformation, and obtain a pose graph using the relative transformation and the similarity score.

[0077] Optionally, in one embodiment of the present application, the registration module 300 is also used to generate a pose graph using each point cloud as a vertex of the pose graph, the relative transformation between two point clouds as an edge, and the similarity score between the two point clouds as the weight of the edge.

[0078] Optionally, in one embodiment of the present application, the registration module 300 includes: a solving unit and a registration unit.

[0079] Among them, the solving unit is used to solve the global transformation matrix of each point cloud.

[0080] The registration unit is used to perform multi-view point cloud registration based on the global transformation matrix.

[0081] It should be noted that the aforementioned explanation of the embodiment of the search-based multi-view point cloud registration method is also applicable to the search-based multi-view point cloud registration device of this embodiment, and will not be repeated here.

[0082] According to the search-based multi-view point cloud registration device proposed in the embodiment of the present application, at least one related point cloud in the remaining point clouds except each point cloud can be searched to generate a search set for each point cloud, thereby constructing a pose graph, and performing multi-view point cloud registration based on the pose graph to restore the complete 3D scene. In this way, a search step is introduced into the multi-view point cloud registration process to significantly remove noise in relative transformations, thereby performing multi-view scene point cloud registration more robustly, and further restoring a complete 3D scene, which is beneficial to the optimization of the pose graph, effectively improving the robustness of the multi-view point cloud registration, and meeting the requirements of multi-view point cloud registration.

[0083] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0084] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0085] When the processor 402 executes the program, the search-based multi-view point cloud registration method provided in the above embodiment is implemented.

[0086] Furthermore, the electronic device further comprises:

[0087] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0088] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0089] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0090] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0092] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0093] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned search-based multi-view point cloud registration method is implemented.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0098] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A search-based multi-view point cloud registration method, It is characterized in that The following steps are involved: Collect multiple disordered point clouds from video data, extract local feature descriptors of each point cloud, and aggregate them into multiple global feature descriptors; Based on the Euclidean distance or cosine similarity between the global feature descriptors, searching for at least one related point cloud in the remaining point clouds except each point cloud to generate a search set for each point cloud; and Constructing a pose graph according to the search set of each point cloud, and performing multi-view point cloud registration based on the pose graph to restore the complete 3D scene; The step of searching at least one related point cloud in the remaining point clouds other than each point cloud based on the Euclidean distance or cosine similarity between the global feature descriptors to generate a search set for each point cloud comprises: Taking any point cloud as a query point cloud, and taking point clouds other than the any point cloud as candidate point clouds; Respectively obtaining global descriptors of the query point cloud and the candidate point cloud, calculating the Euclidean distance or cosine similarity between the global descriptors of the query point cloud and the candidate point cloud, and obtaining a similarity score between the two point clouds; Filtering point clouds having similarity scores greater than a preset threshold, and determining at least one search point cloud for each point cloud; The constructing a pose graph according to the search set of each point cloud comprises: Based on the search set of each point cloud, aligning the query point cloud and the search point cloud in the search set by pairwise registration to obtain a relative transformation; The pose graph is obtained using the relative transformation and the similarity score.

2. The method according to claim 1, It is characterized in that The step of obtaining the pose graph by using the relative transformation and the similarity score includes: The pose graph is generated by using each point cloud as a vertex of the pose graph, using the relative transformation between two point clouds as an edge, and using the similarity score between the two point clouds as the weight of the edge.

3. The method according to any one of claims 1 to 2, It is characterized in that The performing multi-view point cloud registration based on the pose graph includes: Solve the global transformation matrix of each point cloud; Multi-view point cloud registration is performed based on the global transformation matrix.

4. A multi-view point cloud registration device based on search, It is characterized in that include: An aggregation module, used to collect multiple disordered point clouds from video data, extract local feature descriptors of each point cloud, and aggregate them into multiple global feature descriptors; A search module, configured to search for at least one related point cloud in the remaining point clouds except each point cloud based on the Euclidean distance or cosine similarity between the global feature descriptors, and generate a search set for each point cloud; as well as A registration module, configured to construct a pose graph according to the search set of each point cloud, and perform multi-view point cloud registration based on the pose graph to restore a complete 3D scene; The search module includes: A computing unit, configured to use any point cloud as a query point cloud, and use point clouds other than the any point cloud as candidate point clouds, respectively obtain global descriptors of the query point cloud and the candidate point cloud, calculate the Euclidean distance or cosine similarity between the global descriptors of the query point cloud and the candidate point cloud, and obtain a similarity score between the two point clouds; An acquisition unit, used to screen point clouds with similarity scores greater than a preset threshold, and determine at least one search point cloud for each point cloud; The registration module is further used to align the query point cloud and the search point cloud in the search set by pairwise registration based on the search set of each point cloud to obtain a relative transformation, and obtain the pose graph using the relative transformation and the similarity score.

5. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the search-based multi-view point cloud registration method as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, It is characterized in that The program is executed by a processor to implement the search-based multi-view point cloud registration method as described in any one of claims 1 to 3.

Citation Information

Patent Citations

  • A method and system for global registration of three-dimensional point cloud based on feature matching

    CN109087342A

  • Point cloud registration method and device based on multi-feature fusion and storage medium

    CN111340862A

  • Laser point cloud loopback detection method and system suitable for underground roadway

    CN112907491A

  • Global point cloud registration method and system, storage medium and equipment

    CN113344992A

  • Layering-based multi-view point cloud registration method

    CN113920176A