Three-dimensional reconstruction method and related apparatus

Abstract

This application provides a three-dimensional reconstruction method and related apparatus. The three-dimensional reconstruction method includes: acquiring global point cloud data formed by multiple single-frame point cloud data; performing registration operations on the mapping relationship representation of the single-frame point cloud data according to the global point cloud data; generating a target mapping relationship representation based on the registered mapping relationship representation and the mapping relationship representation before registration; and reconstructing the multiple single-frame point cloud data into global point cloud data based on the target mapping relationship representation. The three-dimensional reconstruction method and related apparatus can improve the problem of low accuracy in generating mapping relationship representations caused by the large distance between the measuring device and the tracker. The global point cloud data obtained based on the more accurate mapping relationship representation can more accurately represent the measured object, that is, to a certain extent, improve the accuracy of the global point cloud data generated by the optical tracking three-dimensional scanning system.

Description

Technical Field

[0001] The embodiments described in this application relate to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and related apparatus. Background Technology

[0002] With the rapid development of 3D reconstruction technology, 3D scanners have been widely used in various application fields, including architectural modeling, virtual reality, medical imaging, and industrial inspection. In order to obtain high-precision 3D reconstruction models, external tracking devices are usually used to identify the spatial pose of the 3D scanner, and then the spatial pose and the point cloud data collected by the 3D scanner are combined to generate global point cloud data.

[0003] However, the global point cloud data obtained using the above method still suffers from low accuracy. Summary of the Invention

[0004] In view of this, multiple embodiments of this application aim to provide a three-dimensional reconstruction method and related apparatus, which can improve the accuracy of global point cloud data to a certain extent.

[0005] In a first aspect, one embodiment of this application provides a three-dimensional reconstruction method, comprising: acquiring global point cloud data formed by multiple single-frame point cloud data; performing a registration operation on the mapping relationship representation of the single-frame point cloud data according to the global point cloud data; wherein the mapping relationship representation of the single-frame point cloud data is used to transform the single-frame point cloud data to the coordinate system of the global point cloud data; generating a target mapping relationship representation based on the registered mapping relationship representation and the unregistered mapping relationship representation, and reconstructing the multiple single-frame point cloud data into global point cloud data based on the target mapping relationship representation.

[0006] Optionally, the multiple single-frame point cloud data are raw data collected by the measurement device; the step of obtaining global point cloud data formed by multiple single-frame point cloud data includes: combining multiple single-frame point cloud data based on the mapping relationship representation before registration to form global point cloud data formed by the raw data; and the step of performing registration operation on the mapping relationship representation of the single-frame point cloud data according to the global point cloud data includes: performing registration operation on the single-frame point cloud data based on the global point cloud data formed by the raw data to correct the mapping relationship representation corresponding to the single-frame point cloud data.

[0007] Optionally, the method further includes: registering the mapping relationship representation of the single-frame point cloud data originally acquired by the measurement device based on the reconstructed global point cloud data; and reconstructing the single-frame point cloud data into target global point cloud data based on the registered mapping relationship representation.

[0008] Optionally, the method further includes: performing registration operation again on the mapping relationship representation of the registered single-frame point cloud data based on the reconstructed global point cloud data; and reconstructing the multiple single-frame point cloud data into target global point cloud data based on the re-registered mapping relationship representation.

[0009] Optionally, the step of performing a registration operation on the mapping relationship representation of the single-frame point cloud data based on the global point cloud data includes: matching neighboring points from the global point cloud data that correspond to each point data in the single-frame point cloud data; and performing a registration operation on the mapping relationship representation of the single-frame point cloud data based on each neighboring point.

[0010] Optionally, the method further includes: projecting the standard tracking feature model of the measuring device, represented by the spatial pose of the target mapping relationship, onto the image plane of the tracker; wherein the standard tracking feature model includes multiple tracking feature data for simulating the spatial position of the tracking feature; the projection of the tracking feature data onto the image plane of the tracker forms projected tracking feature data, and the distance between the projected tracking feature data and the image point data corresponding to the tracking feature in the tracking information is used as the projection error; determining multiple matching point data between single-frame point cloud data and global point cloud data; calculating the sum of distance differences between the multiple matching point data of single-frame point cloud data and the multiple matching point data of global point cloud data; and adjusting the target mapping relationship representation so that the value of the projection error and the sum of the distance differences both meet specified conditions.

[0011] Optionally, the method further includes: determining multiple matching point data between single-frame point cloud data and global point cloud data; after performing registration operation on the mapping relationship representation of single-frame point cloud data based on the global point cloud data, if the registration does not converge and the number of multiple matching point data corresponding to the registration operation is greater than a preset value, maintaining the mapping relationship representation before performing the registration operation using the single-frame point cloud data.

[0012] Optionally, the method further includes: calculating the initial distance difference between multiple matching point data of a single frame point cloud data and multiple matching point data of global point cloud data; after performing a registration operation on the mapping relationship representation of the single frame point cloud data based on the global point cloud data, calculating the resulting distance difference between multiple matching point data of the registered single frame point cloud data and multiple matching point data of the global point cloud data; if the resulting distance difference is less than the initial distance difference, retaining the mapping relationship representation corresponding to the single frame point cloud data after the registration operation.

[0013] Secondly, one embodiment of this application also provides a three-dimensional reconstruction apparatus, comprising: an acquisition module for acquiring global point cloud data formed by multiple single-frame point cloud data; a registration module for performing registration operations based on the mapping relationship representation of the single-frame point cloud data to the global point cloud data; wherein the mapping relationship representation of the single-frame point cloud data is used to transform the single-frame point cloud data to the coordinate system of the global point cloud data; and a construction module for generating a target mapping relationship representation based on the registered mapping relationship representation and the unregistered mapping relationship representation, and reconstructing the multiple single-frame point cloud data into global point cloud data based on the target mapping relationship representation.

[0014] Thirdly, one embodiment of this application also provides an electronic device, the electronic device including a memory and a processor, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the three-dimensional reconstruction method as described above.

[0015] Fourthly, one embodiment of this application also provides a computer-readable storage medium storing at least one computer program that, when executed by a processor, can implement the aforementioned three-dimensional reconstruction method.

[0016] Fifthly, one embodiment of this application also provides a computer program product for implementing the three-dimensional reconstruction method as described above.

[0017] In several embodiments provided in this application, global point cloud data generated based on multiple single-frame point cloud data is used to register the mapping relationship representation of the multiple single-frame point cloud data to obtain a more accurate mapping relationship representation. Accordingly, the multiple single-frame point cloud data can be reconstructed into more accurate global point cloud data, that is, the accuracy of the global point cloud data is improved to a certain extent. Attached Figure Description

[0018] Figure 1 A schematic diagram of a three-dimensional reconstruction system provided in one embodiment of this application.

[0019] Figure 2 A schematic diagram of the operation of a three-dimensional reconstruction system provided in one embodiment of this application.

[0020] Figure 3 A flowchart of a three-dimensional reconstruction method provided for one embodiment of this application.

[0021] Figure 4 A schematic diagram of the flying wire defects existing in the 3D reconstruction model constructed for related technologies.

[0022] Figure 5A schematic diagram of tool mark defects in a 3D reconstruction model constructed using related technologies.

[0023] Figure 6 A schematic diagram illustrating the effect of a three-dimensional reconstruction model provided for one embodiment of this application.

[0024] Figure 7 A schematic diagram of a three-dimensional reconstruction apparatus provided in one embodiment of this application.

[0025] Figure 8 A schematic diagram of an electronic device provided according to one embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0027] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0028] 3D reconstruction methods have been widely used in various fields such as architectural modeling, virtual reality, medical imaging, and industrial inspection. Generating 3D models by scanning the surface of an object helps in spatial analysis of that object.

[0029] A 3D reconstruction system can consist of one or more devices used to measure the object being measured and generate a corresponding 3D reconstruction model.

[0030] In one specific embodiment, the 3D scanning system may include a tracker and a measuring device. The measuring device can be used to generate 3D point cloud data for the object being measured, so as to construct a 3D reconstructed model of the object using the 3D point cloud data. Tracking features may be set on the measuring device, and the tracker can be used to track these tracking features to determine the spatial pose of the measuring device based on the tracking features, and to derive the stitched RT (Real Position Measurement) during the scanning process of the object being measured. Specifically, the tracker may include multiple cameras, such as a stereo tracker with two cameras. Generally, the coordinate system of one of the cameras or the central coordinate system of the stereo tracker is used as the base coordinate system to generate the stitched RT. In the stitched RT, Here, Tc refers to the tracker coordinate system, Sc refers to the measurement equipment coordinate system, and S refers to the standard tracking feature model, which can characterize the relative positional relationships between various tracking features on the measurement equipment. Generally, the method for obtaining the stitched RT involves first determining the coordinate system based on the joint fast labeling method. and Furthermore, based on the difference between the tracking feature distribution from the tracker's perspective and the standard tracking feature model, it is determined that... and exist and In situations where it is difficult to be influenced, and The accuracy of this directly affects the accuracy of the stitched RT. Because... and The tracking feature distribution is determined by translating the tracking feature distribution from the tracker's perspective until it is aligned with the standard tracking feature model. Therefore, if the tracking feature distribution from the tracker's perspective is inaccurate, the accuracy of the stitched RT will be affected.

[0031] Specifically, during the scanning of the object being measured, the relative position between the measuring device and the tracker changes. In some cases, when the object being measured is large or a long tracking distance needs to be maintained, the measuring device may move to a position far from the tracker. In this situation, the tracking features tracked by the tracker in the tracker's frame information may decrease, or the number of pixels occupied by the tracked features in the image frame may be small, leading to inaccurate distribution of tracking features from the tracker's perspective. Therefore, when the distance between the tracker and the measuring device is large, the stitched RT obtained after the aforementioned processing will also have significant deviations. Specifically, for example, related technologies may result in defects such as flying lines or tool marks on the surface of the 3D model reconstructed based on global point cloud data. Some solutions can improve the appearance of flying lines or tool marks on the 3D model surface by optimizing hardware performance. However, optimizing hardware performance requires increased investment in hardware, leading to a significant increase in cost, and also requires continuous maintenance of the hardware.

[0032] Furthermore, in related technologies, the above problems can also be solved by optimizing the generated global point cloud data as a whole, such as by filtering the global point cloud data. However, in related technologies, the method of optimizing the global point cloud data as a whole is usually used to optimize high-frequency noise. However, the aforementioned defects such as flying lines or knife marks are manifested as low-frequency noise, and it is difficult to remove low-frequency noise by using the method of optimizing the global point cloud data as a whole. Therefore, the method of optimizing the global point cloud data as a whole is difficult to effectively improve defects such as flying lines or knife marks.

[0033] Therefore, it is necessary to provide a technical solution that can improve the accuracy of stitching RT when converting single-frame point cloud data to the global coordinate system, so as to improve the accuracy of the obtained global point cloud data and improve defects such as surface flying lines or tool marks in the final 3D reconstruction model.

[0034] Please see Figure 1 In various embodiments of this application, the 3D reconstruction system may include a variety of electronic devices. Specifically, the 3D reconstruction system includes a tracker and a measuring device. Both the tracker and the measuring device can integrate computing chips and memory, enabling them to possess certain data processing capabilities. In some embodiments, the electronic devices of the 3D reconstruction system may further include a host computer, which can receive data provided by the tracker and the measuring device and perform data processing. The host computer can be a desktop computer, laptop computer, tablet computer, workstation, or server, etc.

[0035] The measuring equipment may include, but is not limited to, optical measuring equipment and non-optical measuring equipment. Optical measuring equipment may include, but is not limited to, visible light scanners, structured light scanners, laser scanners, and light pens. Non-optical measuring equipment may include, but is not limited to, ultrasonic scanners and X-ray scanners. For example, a laser scanner can measure the distance to an object's surface by emitting a laser beam and detecting changes in the laser's reflection time or phase, thus generating high-precision scan data. Specifically, this scan data can be three-dimensional point cloud data. A structured light scanner can project a structured light pattern (such as stripes or a dot matrix) onto an object's surface and form scan data by detecting the light reflected from the object's surface. In this application, no specific limitations are made on the specific type and principle of the measuring equipment.

[0036] The measuring device possesses tracking features, which can serve as its positioning characteristics. Multiple tracking features can be present and deployed at various locations on the measuring device, allowing for tracking of the device's spatial pose. In some embodiments, the tracking features may include, but are not limited to, marker points, coded points, stereo targets, geometric features of objects, and other features that can be acquired and identified by the tracker. The marker points can be reflective, in which case the tracker emits light and receives the reflected light. In some embodiments, the marker points can be luminescent, in which case the tracker can directly receive the light emitted by the marker points.

[0037] A tracker can be used based on stereo vision tracking principles to output tracking information corresponding to the position and orientation of a measuring device in space. This tracking information can be used to determine pose information representing the spatial orientation of the measuring device. Each tracker includes a camera. The number of cameras in a tracker can be one or more. Preferably, the tracker is a binocular tracker or a multi-view tracker. The number of cameras in different trackers can be the same or different. The tracker can form tracking information from images captured by the cameras. Specifically, the camera can continuously capture multiple image frames, with small time intervals between the multiple image frames, resulting in corresponding differences between the multiple image frames as the measuring device moves in space. The tracking information can include frame information. Each frame information can include image frames captured by multiple cameras of the corresponding tracker at the same time. It can be understood that each frame information can include at least one image frame.

[0038] The 3D reconstruction method can be applied to the aforementioned 3D reconstruction system. Specifically, it can be applied to one or more electronic devices within the system. Those skilled in the art can deploy the electronic devices that execute the 3D reconstruction method according to the specific circumstances.

[0039] Please see Figure 2 and Figure 3 This application provides a three-dimensional reconstruction method. This three-dimensional reconstruction method is applied to an electronic device within a three-dimensional reconstruction system. The three-dimensional reconstruction method may include the following steps.

[0040] Step S110: Obtain global point cloud data formed by multiple single-frame point cloud data.

[0041] Step S120: Perform registration operation based on the mapping relationship representation of the global point cloud data to the single-frame point cloud data; wherein, the mapping relationship representation of the single-frame point cloud data is used to transform the single-frame point cloud data to the coordinate system of the global point cloud data.

[0042] Step S130: Generate a target mapping relationship representation based on the registered mapping relationship representation and the unregistered mapping relationship representation, and reconstruct the multiple single-frame point cloud data into global point cloud data based on the target mapping relationship representation.

[0043] In this embodiment, single-frame point cloud data is data obtained by a measuring device measuring an object at a certain position or from a certain perspective, and mapped to the global coordinate system after an initial mapping relationship. Single-frame point cloud data contains multiple point data corresponding to each point on the surface of the object at the corresponding measurement perspective. Each point data is stored in the form of three-dimensional point cloud data, which can be represented as a matrix. Each row of the matrix represents the coordinates of different point data in the global coordinate system. Before being mapped to the global coordinate system, each point data represents the position of the corresponding point on the surface of the object in the local coordinate system centered on the measuring device. After being mapped to the global coordinate system, each point data represents the position of the corresponding point on the surface of the object in the global coordinate system centered on the tracker. By measuring and mapping the object at multiple positions or from multiple perspectives using the measuring device, multiple single-frame point cloud data corresponding to different measurement perspectives can be obtained.

[0044] In this embodiment, since multiple single-frame point cloud data correspond to different measurement viewpoints, the relative positions between the measurement device and the tracker are different at different measurement viewpoints. The local coordinate system corresponding to the measurement device is also different at different measurement viewpoints. Therefore, each single-frame point cloud data corresponds to a different mapping relationship representation. The mapping relationship representation includes a rotation matrix (R) and a translation matrix (t). The rotation matrix (R) indicates the degree of rotation of the local coordinate system containing the mapping relationship representation relative to the global coordinate system centered on the tracker, and the translation matrix (t) indicates the degree of translation of the local coordinate system containing the mapping relationship representation relative to the global coordinate system centered on the tracker.

[0045] In this embodiment, the mapping relationship representation used to initially generate global point cloud data can be obtained in any of the following ways: determining the pose information provided by the built-in sensor (such as IMU) of the measurement device as the mapping relationship representation; or, estimating the mapping relationship representation by calculating the matching point relationship between two single-frame point cloud data, wherein, since the two single-frame point cloud data are obtained for the same measured object from different measurement perspectives, the two single-frame point cloud data contain point data from different perspectives for the same point on the surface of the measured object, and these two point data can be identified as the matching point between the two single-frame point cloud data; or, obtaining a pre-set rotation matrix (R) and translation matrix (t) as the mapping relationship representation.

[0046] In some implementations, in order to save computing power, the registration operation based on the mapping relationship representation of the single-frame point cloud data according to the global point cloud data can be performed as follows: the global point cloud data is downsampled to obtain the downsampled result, and the mapping relationship representation of the single-frame point cloud data is registered based on the downsampled result.

[0047] In this embodiment, based on the mapping relationships corresponding to multiple single-frame point cloud data, these single-frame point cloud data can be represented in a global coordinate system centered on the tracker and then fused and filtered to form global point cloud data. Global point cloud data can be understood as the result of stitching and fusing multiple single-frame point cloud data, or as a set of points containing multiple 3D coordinates. Since the point cloud data in global point cloud data is relatively sparse and cannot represent the detailed texture of the 3D reconstruction model, model building algorithms (such as Delaunay triangulation) can be used to connect the point cloud data in global point cloud data to form a model composed of triangular meshes. Then, surface reconstruction algorithms (such as Poisson reconstruction) can be used to fill the missing areas in the model, or texture mapping algorithms can be used to enhance the model's expressiveness. However, when the measuring device is far from the tracker, the initially generated mapping relationship may not accurately represent the relationship between the local coordinate system and the global coordinate system. This leads to errors in the result after a single frame of point cloud data is mapped to the global coordinate system. However, when synthesizing global point cloud data, the point data corresponding to the same position in multiple single frame point cloud data are fused or iterated to obtain the correct position of the point data in the global coordinate system. This compensates for the aforementioned errors. Therefore, it can be understood that the global point cloud data at this time has the conditions to serve as the basis for correcting the mapping relationship representation.

[0048] In this embodiment, when the measuring device is far from the tracker, if an initial global point cloud data formed by multiple single-frame point cloud data is used to generate a 3D reconstruction model, the 3D reconstruction model is prone to problems such as... Figure 4 The defects such as flying wires are highlighted by rectangular boxes in the text. Figure 5 The tool mark defect is shown. To address this defect, registration can be performed based on the mapping relationship between global point cloud data and single-frame point cloud data. The registration operation aims to achieve the optimal rotation and translation relationships between the global point cloud data and the single-frame point cloud data through multiple iterations. In simpler terms, the registration operation minimizes the matching point error between the global point cloud data and the single-frame point cloud data, thereby improving... Figure 4 and Figure 5 The defects shown are illustrated. It should be noted that both the global point cloud data and the single-frame point cloud data contain point data for the same point on the surface of the object being measured. Corresponding point data from the global point cloud data and the single-frame point cloud data can be identified as matching points. The registration operation may employ algorithms including, but not limited to: ICP algorithm, NDT algorithm, 3DSC algorithm, PFH algorithm, FPFH algorithm, etc.

[0049] In this embodiment, taking the ICP algorithm as an example, the registration operation can be performed as follows: Initial mapping relationship representation = (R0, t0); for each point data p in a single frame of point cloud data i Traverse the global point cloud data to determine the nearest global point data q. i Thus, we obtain the set of point pairs {(p i ,q i Furthermore, using the least squares method of rigid transformation, the rotation matrix (R1) and translation matrix t(1) in the mapping relation representation are calculated, such that p i The mapping relationship represents the transformation as close as possible to q. i The least squares representation of a rigid transformation is E(R,t)=∑ i ‖Rp i +t―q i || 2 Minimizing E(R,t) yields the required rotation matrix (R2) and translation matrix (t2), enabling registration of global point cloud data and single-frame point cloud data. The resulting registered rotation matrix (R2) and translation matrix (t2) can be used to reconstruct the global point cloud data. In some implementations, the target mapping relationship is represented by: the registered rotation matrix (R2) and the registered translation matrix (t2), where the registered rotation matrix (R2) = R1R0 and the registered translation matrix (t2) = R2 t0 + t1. Specifically, substituting the registered rotation matrix (R2) and translation matrix (t2) into the expression P′ = RP + t yields the reconstructed global point cloud data P′.

[0050] In this embodiment, the registration operation can be performed multiple times to minimize the error between the reconstructed global point cloud data and the single-frame point cloud data. The 3D reconstruction model generated based on the final global point cloud data is as follows: Figure 6 As shown, it overcomes Figure 4 and Figure 5 The displayed defects are addressed by improving the quality of the generated 3D reconstruction model. It can be understood that in this embodiment, during the generation of global point cloud data, based on a more accurate mapping representation, the registered single-frame point cloud data undergoes fusion and filtering processes to obtain the reconstructed global point cloud data.

[0051] In this embodiment, global point cloud data generated from multiple single-frame point cloud data is used to register the mapping relationship representation of the multiple single-frame point cloud data, resulting in a more accurate mapping relationship representation. This improves the problem of low accuracy in mapping relationship representation generation caused by the large distance between the measuring device and the tracker. Based on the more accurate mapping relationship representation, multiple single-frame point cloud data can be reconstructed into more accurate global point cloud data. The resulting global point cloud data can more accurately represent the measured object. This also solves the problem of defects such as flying lines and tool marks on the surface of the 3D reconstructed model caused by the low accuracy of the global point cloud data.

[0052] In some implementations, the 3D reconstruction method may further include: acquiring real-time point data measured by a measuring device at a specified location; representing the real-time point data in a global coordinate system based on a mapping relationship corresponding to the specified location; and performing real-time registration operations on the mapping relationship corresponding to the specified location based on a portion of the global data in the global coordinate system to obtain a new mapping relationship. This new mapping relationship is used to update the real-time portion of the global data, thereby achieving real-time updates to the global point cloud data. The 3D reconstruction model obtained through this real-time update method has higher precision. Furthermore, because this process is implemented in real-time, the user terminal can display the real-time reconstruction process of the 3D reconstruction model while ensuring that the displayed 3D reconstruction model possesses the appropriate level of precision.

[0053] In some implementations, the 3D reconstruction apparatus can combine multiple single-frame point cloud data to form global point cloud data from the original data based on the mapping relationship representation before registration; and the 3D reconstruction apparatus can perform registration operations on the single-frame point cloud data based on the global point cloud data formed from the original data to correct the mapping relationship representation corresponding to the single-frame point cloud data.

[0054] In this embodiment, to avoid error accumulation, the 3D reconstruction device performs registration operations based on global point cloud data formed from the original data. This avoids the problem of error accumulation caused by insufficient accuracy of the mapping relationship representation. The original data refers to single-frame point cloud data in the scanner coordinate system obtained directly by the measuring equipment, or single-frame point cloud data transformed from the scanner coordinate system to the global point cloud coordinate system based on the mapping relationship representation. For each single-frame point cloud data, the 3D reconstruction device registers it with the global point cloud data formed from the original data, thereby obtaining a mapping relationship representation corresponding to multiple single-frame point cloud data. Each mapping relationship representation can be used to reconstruct the global point cloud data to improve its accuracy, which in turn helps improve the quality of the 3D reconstruction model generated based on the global point cloud data.

[0055] In some implementations, the 3D reconstruction device can register the mapping relationship representation of the single-frame point cloud data originally acquired by the measurement device based on the reconstructed global point cloud data; and reconstruct the single-frame point cloud data into target global point cloud data based on the registered mapping relationship representation.

[0056] In this embodiment, to further improve the accuracy of global point cloud data generation, the 3D reconstruction device can register the mapping representation of the original single-frame point cloud data acquired by the measurement equipment based on the reconstructed global point cloud data, thereby correcting the mapping representation again. Based on the corrected mapping representation, the original single-frame point cloud data can be reconstructed into target global point cloud data. The target global point cloud data has higher accuracy than the previously reconstructed global point cloud data, which can further improve the model quality of the 3D reconstruction model generated based on the global point cloud data.

[0057] In some implementations, the 3D reconstruction device can perform registration operations again on the mapping relationship representation of the registered single-frame point cloud data based on the reconstructed global point cloud data; and reconstruct the multiple single-frame point cloud data into target global point cloud data based on the re-registered mapping relationship representation.

[0058] In this embodiment, in order to further improve the accuracy of global point cloud data generation, the 3D reconstruction device can re-register the registered mapping relationship representation based on the reconstructed global point cloud data to further improve the accuracy of the mapping relationship representation. Based on the re-registered mapping relationship representation, multiple single-frame point cloud data can be reconstructed into target global point cloud data with higher accuracy, thereby further improving the model quality of the 3D reconstruction model generated based on the global point cloud data.

[0059] In some implementations, the 3D reconstruction device can match neighboring points from the global point cloud data to each point in the single frame point cloud data; and perform registration operations based on the mapping relationship between each neighboring point and the single frame point cloud data.

[0060] In this embodiment, a single frame of point cloud data contains multiple point data. Traversing the global point cloud data can match the nearest points corresponding to each point data. The nearest point has the smallest distance to its corresponding point data. The nearest points corresponding to each point data can be different. Based on each nearest point data, the RT corresponding to the single frame of point cloud data can be iterated to obtain the registered mapping relationship representation. Representing each point data in the single frame of point cloud data in the global coordinate system based on the registered mapping relationship can minimize the sum of the distances between each point data in the single frame of point cloud data and its corresponding nearest points. The target mapping relationship representation (i.e., the aforementioned R2t2) further generated based on the registered mapping relationship representation has higher accuracy, which is beneficial for optimizing the model quality of the 3D reconstruction model.

[0061] In some embodiments, the 3D reconstruction device can project the standard tracking feature model of the measuring device, represented by the spatial pose of the target mapping relationship, onto the image plane of the tracker; wherein, the standard tracking feature model includes multiple tracking feature data for simulating the spatial position of the tracking feature; the projection of the tracking feature data onto the image plane of the tracker forms projected tracking feature data, and the distance between the projected tracking feature data and the image point data corresponding to the tracking feature in the tracking information is used as the projection error; multiple matching point data between single-frame point cloud data and global point cloud data are determined; the sum of distance differences between the multiple matching point data of single-frame point cloud data and the multiple matching point data of global point cloud data is calculated; the target mapping relationship representation is adjusted so that the value of the projection error and the sum of the distance differences both meet the specified conditions.

[0062] In this embodiment, the measuring device corresponds to a standard tracking feature model, which includes tracking feature data corresponding to the tracking features. It can be understood that the standard tracking feature model can carry the spatial position information of the tracking features of the measuring device. Since there is a fixed relationship between the measuring device and the tracking features, the measuring device can be characterized by the standard tracking feature model. Based on this, the standard tracking feature model of the measuring device, represented by a target mapping relationship, can be projected onto the image plane of the tracker, so that the tracking feature data in the standard tracking feature model represents the attitude of the measuring device in two-dimensional form. Here, the image plane of the tracker refers to the two-dimensional plane observed from the tracker's perspective. Furthermore, by projecting the tracking feature data onto the image plane to form projected tracking feature data and image point data corresponding to the tracking features in the tracking information, the distance between the two can be determined as the projection error. The larger the distance between them, the larger the error in representing the target mapping relationship. Furthermore, multiple matching point pairs exist between single-frame point cloud data and global point cloud data. When the multiple matching point data mapped from the single-frame point cloud data based on the mapping relationship can be aligned with the corresponding matching point data in the global point cloud data in the global coordinate system, it indicates that the target mapping relationship is accurate. Therefore, the target mapping relationship representation corresponding to the single-frame point cloud data is adjusted to minimize the sum of distance differences and projection errors between the multiple matching point data in the single-frame point cloud data and the multiple matching point data in the global point cloud data. The minimized sum of distance differences indicates that the single-frame point cloud data can be accurately mapped to the global coordinate system. Specific conditions are used to constrain the values ​​of the sum of distance differences and projection errors to achieve the predetermined 3D reconstruction accuracy.

[0063] In some embodiments, the 3D reconstruction device may determine a plurality of matching point data between single-frame point cloud data and global point cloud data; after performing registration operations on the mapping relationship representation of the single-frame point cloud data based on the global point cloud data, when the registration does not converge and the number of the plurality of matching point data corresponding to the registration operation is greater than a preset value, the mapping relationship representation before performing the registration operation using the single-frame point cloud data is maintained.

[0064] In this embodiment, after the 3D reconstruction device performs registration operations on the mapping relationship representation of the single-frame point cloud data based on the global point cloud data, it calculates the root mean square of the distances of the matching point pairs between the single-frame point cloud data and the global point cloud data as the error err, and records the error err obtained in the first calculation as err0. If err is less than or equal to err0 and |final_err - err| < N1, the registration count is incremented; otherwise, the registration count is set to 0. Here, the registration count refers to the number of registrations that meet the expectations, final_err is a preset error, and N1 is a preset value, such as 0.03, 0.04, 0.05... 0.08.

[0065] When the registration count >= N2, it is determined that the registration converges and the mapping relationship representation corresponding to the single-frame point cloud data after the registration operation is retained. N2 is a preset value, such as 2, 3, 4... 10.

[0066] When the registration count < N2, err is assigned to final_err, and the iteration count is incremented, and then the above steps are looped. The maximum value of the iteration count is limited to N3, and N3 is a preset value, such as 25, 26, 27, 30... 41.

[0067] When both the rotation angle and the translation value in the latest registration operation result meet the corresponding preset conditions, it is determined that the registration converges and the mapping relationship representation corresponding to the single-frame point cloud data after the registration operation is retained.

[0068] In some embodiments, after the registration iteration ends, if the rotation angle or translation value in the latest registration operation result does not meet the corresponding preset condition, and the number of registration times < N2, then it is determined at this time that the registration has not converged. If the number of multiple matching point data corresponding to the first registration operation is greater than the preset value, it means that the number of matching point pairs that can be aligned between the global point cloud data and the single-frame point cloud data decreases, that is, it means that the accuracy of the mapping relationship obtained by registration decreases, and the single-frame point cloud data cannot be accurately mapped to the global coordinate system. In this case, the mapping relationship representation before performing the registration operation using the single-frame point cloud data can be maintained to avoid the decrease in the accuracy of the global point cloud data. If the number of multiple matching point data corresponding to the first registration operation is less than or equal to the preset value, the single-frame point cloud data is discarded. Among them, the preset value is calculated based on the number of point data in the corresponding single-frame point cloud data combined with a preset ratio (such as 50%).

[0069] In some embodiments, the three-dimensional reconstruction device can calculate the initial distance difference between multiple matching point data of the single-frame point cloud data and multiple matching point data of the global point cloud data; after performing a registration operation on the mapping relationship representation of the single-frame point cloud data according to the global point cloud data, calculate the result distance difference between multiple matching point data of the registered single-frame point cloud data and multiple matching point data of the global point cloud data; in the case where the result distance difference is less than the initial distance difference, retain the mapping relationship representation corresponding to the single-frame point cloud data after the registration operation.

[0070] In this embodiment, since there are corresponding matching point pairs between multiple point data of the single-frame point cloud data and multiple point data of the global point cloud data, the initial distance difference can be calculated for the corresponding matching point pairs between the two. After performing the registration operation, if the distance difference between the corresponding matching points between the registered single-frame point cloud data and the global point cloud data is less than the initial distance difference, it is determined that the result of the registration operation meets the expectation. Therefore, the mapping relationship representation after the registration operation can be used as the basis for reconstructing the global point cloud data to improve the accuracy of the global point cloud data. Among them, specifically, the distance difference can be any one of the mean difference, sum difference, variance difference, etc., and the embodiments of the present application do not limit this.

[0071] Please refer to Figure 7One embodiment of this application also provides a three-dimensional reconstruction apparatus. The three-dimensional reconstruction apparatus may include: an acquisition module for acquiring global point cloud data formed by multiple single-frame point cloud data; a registration module for performing registration operations based on the mapping relationship representation of the single-frame point cloud data to the global point cloud data; wherein the mapping relationship representation of the single-frame point cloud data is used to transform the single-frame point cloud data to the coordinate system of the global point cloud data; and a construction module for generating a target mapping relationship representation based on the registered mapping relationship representation and the unregistered mapping relationship representation, and reconstructing the multiple single-frame point cloud data into global point cloud data based on the target mapping relationship representation.

[0072] In this embodiment, the specific functions and effects of the three-dimensional reconstruction system can be explained by referring to other embodiments of this application, and will not be repeated here.

[0073] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to implement the three-dimensional reconstruction method as described above.

[0074] This application also provides a computer program product containing instructions that, when executed by a processor, implement the three-dimensional reconstruction method as described above.

[0075] Please see Figure 8 This application provides an electronic device comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, which, when executed by the one or more processors, enable the one or more processors to implement the aforementioned three-dimensional reconstruction method.

[0076] In some embodiments, the electronic device may include a processor, a storage medium, and a communication interface connected to a system bus. The storage medium may store related computer programs.

[0077] The user information or user account information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, etc.) involved in various embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws and regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0078] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments of this application, and are not intended to limit the scope of the invention.

[0079] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0080] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and the implementation methods in this application are not limited in this respect.

[0081] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0082] It is understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0083] It is understood that the memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0084] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0088] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0089] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: Acquire global point cloud data formed by multiple single-frame point cloud data; Registration is performed based on the mapping relationship between the global point cloud data and the single-frame point cloud data; wherein, the mapping relationship representation of the single-frame point cloud data is used to transform the single-frame point cloud data to the coordinate system of the global point cloud data; A target mapping representation is generated based on the registered mapping representation and the unregistered mapping representation, and the multiple single-frame point cloud data are reconstructed into global point cloud data based on the target mapping representation.

2. The method according to claim 1, characterized in that, The multiple single-frame point cloud data are the raw data collected by the measurement device; The steps for obtaining global point cloud data formed by multiple single-frame point cloud data include: combining multiple single-frame point cloud data based on the mapping relationship representation before registration to form global point cloud data formed by the original data; The step of performing registration operations on the mapping relationship representation of the single-frame point cloud data based on the global point cloud data includes: Based on the global point cloud data formed from the original data, a registration operation is performed on the single-frame point cloud data to correct the mapping relationship representation corresponding to the single-frame point cloud data.

3. The method according to claim 1, characterized in that, The method further includes: Based on the reconstructed global point cloud data, the mapping relationship of the original single-frame point cloud data acquired by the measurement equipment is registered. Based on the registered mapping relationship, the single-frame point cloud data is reconstructed into the target global point cloud data.

4. The method according to claim 1, characterized in that, The method further includes: Based on the reconstructed global point cloud data, registration operation is performed again on the mapping relationship representation of the registered single-frame point cloud data; Based on the mapping relationship after re-registration, the multiple single-frame point cloud data are reconstructed into target global point cloud data.

5. The method according to claim 1, characterized in that, The steps of performing registration operations based on the mapping relationship representation of the global point cloud data to the single-frame point cloud data include: Match the nearest points from the global point cloud data to the point data in the single frame point cloud data; Registration operations are performed based on the mapping relationship between each neighboring point and the single-frame point cloud data.

6. The method according to claim 1, characterized in that, Also includes: The standard tracking feature model of the measuring device, represented by the spatial attitude of the target mapping relationship, is projected onto the image plane of the tracker; wherein, the standard tracking feature model includes multiple tracking feature data for simulating the spatial position of the tracking feature; the projection of the tracking feature data onto the image plane of the tracker forms projected tracking feature data, and the distance between the projected tracking feature data and the image point data corresponding to the tracking feature in the tracking information is used as the projection error; Determine multiple matching point data between single-frame point cloud data and global point cloud data; Calculate the sum of distance differences between multiple matching point data in a single frame of point cloud data and multiple matching point data in global point cloud data; The target mapping relationship is adjusted so that the value of the projection error and the sum of the distance differences both meet the specified conditions.

7. The method according to claim 1, characterized in that, The method further includes: Determine multiple matching point data between single-frame point cloud data and global point cloud data; After performing registration operations on the mapping relationship representation of single-frame point cloud data based on the global point cloud data, if the registration does not converge and the number of multiple matching point data corresponding to the registration operation is greater than a preset value, the mapping relationship representation before performing the registration operation using the single-frame point cloud data is maintained.

8. The method according to claim 7, characterized in that, The method further includes: Calculate the initial distance difference between multiple matching point data in a single frame of point cloud data and multiple matching point data in global point cloud data; After performing a registration operation on the mapping relationship representation of the single-frame point cloud data based on the global point cloud data, the distance difference between the registered single-frame point cloud data and the global point cloud data is calculated. If the resulting distance difference is less than the initial distance difference, the mapping relationship representation corresponding to the single-frame point cloud data after the registration operation is retained.

9. A three-dimensional reconstruction device, characterized in that, include: The acquisition module is used to acquire global point cloud data formed by multiple single-frame point cloud data. The registration module is used to perform registration operations based on the mapping relationship representation of the global point cloud data to the single-frame point cloud data; wherein, the mapping relationship representation of the single-frame point cloud data is used to transform the single-frame point cloud data to the coordinate system of the global point cloud data; A construction module is used to generate a target mapping relationship representation based on the registered mapping relationship representation and the unregistered mapping relationship representation, and to reconstruct the multiple single-frame point cloud data into global point cloud data based on the target mapping relationship representation.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores at least one computer program, which is loaded and executed by the processor to implement the three-dimensional reconstruction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, is capable of implementing the three-dimensional reconstruction method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product is used to implement the three-dimensional reconstruction method as described in any one of claims 1 to 8.

Images