Low-cost three-dimensional mapping method and system based on intelligent handheld mobile terminal

Abstract

The application relates to a low-cost three-dimensional mapping method, system, electronic equipment and storage medium based on a smart handheld mobile terminal, which collects data by using the smart mobile terminal, a civil equipment, and realizes alignment of each track in the Z-axis direction by constructing an elevation map based on the data, and finally performs three-dimensional reconstruction by using the aligned data. The smart mobile terminal, a handheld equipment, can be used to collect data on the ground in a good GPS signal and realize low-cost fast mapping.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction, and in particular to a low-cost 3D mapping method, system, computer device, and computer-readable storage medium based on a smart handheld mobile terminal. Background Technology

[0002] In the field of Augmented Reality (AR), AR content can be broadly divided into two categories: environment-related and environment-independent. Environment-related content can be further divided into real-time environment reconstruction and pre-reconstruction of the environment.

[0003] Real-time environment reconstruction utilizes the SLAM system of AR devices (such as AR glasses and mobile phones) to construct the surrounding environment in real time, enabling low-cost interaction with the physical world. However, due to limitations in the device's computing power and storage, the area of ​​the map constructed in real time is limited. For specific large-area scenes (such as amusement parks and museums), pre-construction of the environment is required.

[0004] In pre-environment reconstruction methods in related technologies, GPS information from the data source collected by handheld mobile terminals can only provide XY constraints, resulting in the reconstruction results not being aligned in the Z-axis (vertical) direction, thus affecting the accuracy of the reconstruction results. While using more accurate real-time kinematic (RTK) technology can provide Z-direction constraints, the mapping cost also increases. Summary of the Invention

[0005] This application provides a low-cost 3D mapping method, system, computer device, and computer-readable storage medium based on a smart handheld mobile terminal, to at least solve the problems of low cost and poor convenience in 3D reconstruction using RTK information-assisted alignment in related technologies.

[0006] In a first aspect, embodiments of this application provide a low-cost 3D mapping method based on a smart handheld mobile terminal, the method comprising:

[0007] Acquire initial mapping data collected through a smart handheld mobile terminal, wherein the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes an image and gravity-aligned pose and GPS positioning information corresponding to the image;

[0008] Using the GPS positioning information as constraint information, the initial mapping data is optimized using 3DOF, and the horizontal direction and heading angle of each trajectory are aligned to obtain the initial alignment data in X, Y, pitch, roll and yaw relative to the geographic coordinate system.

[0009] An elevation map is constructed based on the initial alignment data. The elevation map is then optimized using a maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, resulting in aligned data where all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data is a relative value.

[0010] Three-dimensional scene reconstruction is performed based on the alignment data.

[0011] In some embodiments, the method further includes:

[0012] For each trajectory in the initial alignment data, feature point matching and triangulation of the image are performed to obtain the three-dimensional point cloud information of each trajectory in the XY space.

[0013] Based on the three-dimensional point cloud information, visual loop closure detection is performed on each trajectory to obtain the relative pose transformation of each trajectory relative to other target trajectories that overlap and match with it in the XY space.

[0014] Based on the relative pose transformation, the altitude difference between each trajectory and the other target trajectories is obtained.

[0015] In some embodiments, constructing an elevation map based on the initial alignment data includes:

[0016] Obtain the number of successful loop closures for each trajectory in the loop closure detection;

[0017] The elevation graph is constructed by using the elevation values ​​of each trajectory as nodes, the elevation difference between any two trajectories with loops as edges, and the number of successful loops corresponding to the elevation difference as the weight of the edges.

[0018] In some embodiments, the elevation map is optimized using a maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data, resulting in aligned data including:

[0019] The maximum spanning tree algorithm is applied to the elevation map to obtain the maximum spanning tree model corresponding to the elevation values ​​between nodes;

[0020] The root node of the maximum spanning tree model is determined, and the elevation value is propagated based on the maximum spanning tree model to calculate the elevation difference between other nodes and the root node, wherein the elevation value of the root node is set to 0.

[0021] Based on the elevation difference between each node and the root node, the elevation values ​​of all nodes are aligned to obtain aligned data in the geographic coordinate system, where all six degrees of freedom are aligned.

[0022] In some embodiments, the elevation values ​​of each trajectory in the initial alignment data are aligned to the same coordinate system. After obtaining the alignment data, multiple trajectories are aligned to the geographic coordinate system in six degrees of freedom: X, Y, Z, Pitch, Roll, and Yaw. The method further includes:

[0023] Based on the poses of the successfully looped images under different trajectories during the visual loop closure detection process, the pose of the virtual reference frame is determined. The orientation of the virtual reference frame pose is the orientation of the image under its original trajectory after 3DOF optimization, and the position of the virtual reference frame pose is the average of the pose translation of the image under the original trajectory and all successfully looped trajectories.

[0024] With the virtual reference frame pose fixed, the alignment data is optimized using 6DOF to obtain the optimized alignment data.

[0025] Based on the optimized aligned data, a 3D reconstruction is performed to obtain a 3D model.

[0026] In some embodiments, determining the pose of a virtual reference frame based on the pose of any image across multiple trajectories includes:

[0027] The pose of the virtual reference frame corresponding to image M in Trajk is calculated using the following formula;

[0028] Translation_M_to_Geo(mean)=(Translation_M_to_Geo(Traj1)+Translation_M_to_Geo(Traj2)+……+Translation_M_to_Geo(Trajn)) / n;

[0029] Wherein, Translation_M_to_Geo(Traj1), Translation_M_to_Geo(Traj2), and Translation_M_to_Geo(Trajn) are the positions of image M in the geographic coordinate system calculated under trajectory 1, trajectory 2, ..., trajectory n, respectively; Rotation_M_to_Geo(Trajk) is the orientation of image M in the geographic coordinate system under the 3DOF optimized trajectory K; and Translation_M_to_Geo(mean) and Rotation_M_to_Geo(Trajk) constitute the pose of the virtual reference frame corresponding to image M in Trajk.

[0030] Secondly, embodiments of this application provide a low-cost 3D mapping system based on a smart handheld mobile terminal. The system includes: an acquisition module, an alignment module, and a reconstruction module, wherein:

[0031] The acquisition module is used to acquire initial mapping data collected by a smart handheld mobile terminal. The initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes an image and gravity-aligned pose and GPS positioning information corresponding to the image.

[0032] The optimization module is used to perform 3DOF optimization on the initial mapping data with the GPS positioning information as a constraint, align the horizontal direction and heading angle of each trajectory, and obtain the initial alignment data that is aligned in X, Y, pitch, roll and yaw relative to the geographic coordinate system.

[0033] Furthermore, an elevation map is constructed based on the initial alignment data, and the elevation map is optimized by the maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, thereby obtaining aligned data in which all six degrees of freedom are aligned on the geographic coordinate system, wherein the Z-axis data in the aligned data is a relative value;

[0034] The reconstruction module is used to reconstruct a 3D scene based on the alignment data.

[0035] In some embodiments, the system further includes an image algorithm module, wherein the image algorithm module is used for:

[0036] For each trajectory in the initial alignment data, feature point matching and triangulation of the image are performed to obtain the three-dimensional point cloud information of each trajectory in the XY space.

[0037] Based on the three-dimensional point cloud information, visual loop closure detection is performed on each trajectory to obtain the relative pose transformation of each trajectory relative to other target trajectories that overlap and match with it in the XY space.

[0038] Based on the relative pose transformation, the altitude difference between each trajectory and the other target trajectories is obtained.

[0039] Thirdly, embodiments of this application provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0041] Compared to related technologies, the low-cost 3D mapping method based on a smart handheld mobile terminal provided in this application utilizes the SALM algorithm collected by the smart mobile terminal to collect gravity scene data on the pitch and roll. Furthermore, under the constraint of GPS information, 3DOF optimization is performed to align the original data in the X and Y directions and yaw. Finally, based on the data aligned in the geographic coordinate system in X, Y, pitch, roll and yaw, an elevation map is constructed to align each trajectory in the Z-axis direction, and 3D reconstruction is performed using the data aligned in the geographic coordinate system.

[0042] Furthermore, considering the inherent absolute error of GPS and the cumulative relative error of each trajectory, which leads to pose errors of the same image across different trajectories, this application, after obtaining data that is theoretically aligned to the geographic coordinate system across all six degrees of freedom, further calculates the average pose of the same image across different trajectories to obtain a virtual reference frame pose. This virtual reference frame pose is used to describe the true pose of the same image in geographic coordinates as closely as possible, thereby eliminating the differences of the same image across different trajectories and improving the accuracy of the 3D reconstruction results.

[0043] To address the issues of ordinary GPS information not providing usable altitude values ​​and varying altitudes between multiple tracks, this solution enables the use of a smart mobile terminal to collect data on ground with good GPS signals and achieve rapid mapping at low cost. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1 This is a flowchart of a low-cost 3D mapping method based on a smart handheld mobile terminal according to an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of a 3DOF optimization according to an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the trajectory pose before elevation alignment according to the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of the trajectory pose after elevation alignment according to an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of a trajectory pose optimized by 6DOF according to another embodiment of this application;

[0050] Figure 6 This is a schematic diagram illustrating one method of obtaining a virtual reference frame according to an embodiment of this application;

[0051] Figure 7 This is a schematic diagram illustrating 6DOF optimization under the premise of a fixed virtual reference frame pose, according to an embodiment of this application.

[0052] Figure 8 This is a structural block diagram of a low-cost 3D mapping system based on a smart handheld mobile terminal, according to an embodiment of this application.

[0053] Figure 9 This is a schematic diagram of another low-cost 3D mapping system based on a smart handheld mobile terminal according to an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0056] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0057] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0058] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0059] In the field of 3D reconstruction, pre-environment reconstruction can utilize different methods, which can be broadly categorized based on the data source acquisition equipment:

[0060] 1. Static mapping equipment (such as a static LiDAR scanner equipped with multiple cameras)

[0061] a. Advantages: High accuracy and short mapping time;

[0062] b. Disadvantages: High equipment cost, long data collection time, and high professional requirements for data collection personnel;

[0063] 2. Mobile surveying equipment (such as handheld dynamic lidar scanners equipped with cameras)

[0064] a. Advantages: High accuracy, short mapping time, and short data acquisition time.

[0065] b. Disadvantages: Equipment costs are relatively high, and data collection personnel require a certain level of expertise.

[0066] 3. Civil aviation equipment (such as drones)

[0067] a. Advantages: High accuracy, short data acquisition time

[0068] b. Disadvantages: It cannot be used with ground-based AR devices; it can only be used as supplementary map data from a bird's-eye view perspective.

[0069] 4. Consumer handheld devices (such as panoramic action cameras, smartphones, etc.)

[0070] a. Advantages: Low equipment cost, relatively short data collection time, and minimal training required for data collection personnel.

[0071] b. Disadvantages: Relatively low accuracy and long mapping time.

[0072] In the data sources collected by handheld devices, GPS information can only provide XY constraints, which makes it impossible to align the reconstruction results in the Z-axis (vertical) direction. While using more accurate real-time kinematic (RTK) technology can provide Z-direction constraints, the acquisition cost and the professional skills required of personnel also increase.

[0073] In view of this, this application provides a solution for providing "Visual Positioning Service" (VPS) for pre-reconstructed environmental content. This solution can use handheld devices such as "smartphones" to collect and reconstruct map data of multiple trajectories in outdoor environments with good GPS signals (multi-device collaborative mapping and positioning). This significantly reduces mapping time while retaining the advantages of traditional mapping techniques.

[0074] Figure 1 This is a flowchart of a low-cost 3D mapping method based on a smart handheld mobile terminal according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0075] S101, acquire initial mapping data collected by a smart handheld mobile terminal, wherein the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes the gravity alignment pose and GPS positioning information corresponding to the image.

[0076] The intelligent handheld mobile terminal can be, but is not limited to, a smartphone, tablet computer, or other civilian device. Furthermore, the trajectory consists of a set of continuous images and the relative poses of each image in local coordinates.

[0077] In one exemplary embodiment, a person uses a smartphone to collect the aforementioned trajectory in a low-cost, efficient and convenient manner by deploying and running a SLAM algorithm. Specifically, the program or application that supports running the SLAM algorithm can be ARKit on iOS or ARCore on Android.

[0078] In addition, the collected tuple data corresponds to real-world offline scenarios, such as exhibition halls, shopping malls, and scenic spots.

[0079] It should be noted that, considering that the main focus of this application is to use GPS information that cannot provide Z-axis constraints to achieve alignment of different trajectories in the Z-axis direction, the actual collected multiple trajectories should cover different heights in the real scene as much as possible, such as different floors of a shopping mall.

[0080] Furthermore, the pose acquired by the SLAM algorithm is itself gravity aligned, which means rotating the Z-axis in the SLAM world coordinate system to align with the direction of gravity. This implies that the object's roll and pitch angles have been corrected relative to the direction of gravity, i.e., pitch and roll are aligned, but yaw may not be aligned.

[0081] Additionally, it should be noted that gravity alignment is not the same as Z-axis alignment. Gravity alignment typically refers to aligning an object or coordinate system's axis (such as the Z-axis) with the direction of gravity (i.e., the direction of Earth's gravitational pull) in physical space. This alignment method focuses primarily on direction, rather than specific location or altitude.

[0082] In this embodiment, elevation alignment is a concept related to geographic coordinate systems. It refers to aligning the height of a location or object numerically with a reference plane. In this case, elevation is a specific numerical value representing the height of the location or object relative to the reference plane.

[0083] In addition, the number of collected trajectories can be flexibly determined according to the area and complexity of the scene, and this embodiment does not limit this; however, it should be understood that the more data there is and the richer the sample types are, the more accurate the alignment results will be, and the more accurate the 3D reconstruction results will be for positioning.

[0084] S102, using GPS positioning information as constraint information, perform 3DOF optimization on the initial mapping data, align the horizontal direction and heading angle of each trajectory, and obtain the initial alignment data in X, Y, pitch, roll and yaw relative to the geographic coordinate system.

[0085] It is understandable that the data obtained in step S101 are in their respective local coordinate systems. Although they are gravity aligned (i.e., pitch and roll are aligned), their XY and yaw are not aligned, and their elevations are not aligned either.

[0086] Figure 2 This is a schematic diagram of a 3DOF optimization according to an embodiment of this application, such as... Figure 2 As shown, in this step, the XY plane of multiple trajectories is optimized by the 3DOF optimization module so that each trajectory is aligned with the geographic coordinate system in the horizontal direction.

[0087] Specifically, the 3DOF optimization process can align each trajectory to the geographic coordinate system horizontally using the following formula:

[0088]

[0089] Where Geo is the variable to be optimized, i.e., the pose of the image aligned to the geographic coordinate system; GPS represents the constraints provided by GPS positioning information; Track is the pose in the local coordinate system; and w k These are weighting coefficients used to eliminate incorrect GPS constraints; the minus sign "-" in the formula for calculating Pose means calculating the difference between two Pose matrices, where the inverse of the Pose matrix to the left of the minus sign is multiplied by the Pose matrix to the right of the minus sign.

[0090] It is the constraint between the XY aligned pose in geographic coordinates and the relative pose in the trajectory acquired by the SLAM algorithm, || GPS k -Geo k || 2 It refers to the constraint between GPS positioning information and the pose to be optimized. It is used to eliminate erroneous GPS constraints and updates w using the Expectation-Maximization algorithm. k .

[0091] In summary, this optimization formula takes into account the relative constraints between trajectories, the constraints between GPS positioning information and the actual geographic pose. The goal of this formula is to achieve 3DoF optimization by minimizing the error under constraints, and to align different trajectories on the X-axis, Y-axis and yaw.

[0092] S103. Construct an elevation map based on the initial alignment data, optimize the elevation map using the maximum spanning tree, align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, and obtain aligned data in which all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data is a relative value. Perform 3D scene reconstruction based on the aligned data.

[0093] It should be noted that some preprocessing steps need to be performed before this step to provide basic information for constructing the elevation map. Specifically, these steps include the following:

[0094] Step 1: For each trajectory in the initial alignment data, perform feature point matching and triangulation of the image to obtain the three-dimensional point cloud information of each trajectory in the XY space.

[0095] Step 2: Based on the 3D point cloud information, perform visual loop closure detection on each trajectory to obtain the relative pose transformation of each trajectory relative to other target trajectories that overlap and match with it in the XY space.

[0096] Step 3: Based on the relative pose transformation, obtain the altitude difference between each trajectory and other target trajectories.

[0097] It should be noted that the above image matching process involves global and local feature extraction as well as matching methods. The specific implementation method of this step is not limited in the embodiments of this application.

[0098] Global features can be achieved using traditional algorithms such as DBOW, or deep learning algorithms such as NetVLAD and MixVPR. Local features can be achieved using traditional algorithms such as SIFT, or deep learning algorithms such as SuperPoint. Feature point matching can be achieved using traditional algorithms such as RANSAC+FH geometric verification, or deep learning algorithms such as SuperGlue.

[0099] Furthermore, since the raw data collected in S101 is already gravity-aligned (i.e., Pitch and Roll are already aligned), after 3DoF optimization under GPS constraints in step S202, each trajectory is aligned in the X and Y directions and in the Yaw direction. At this point, each trajectory in the raw data is already aligned in five degrees of freedom: X, Y, Pitch, Roll, and Yaw. In this step, by constructing an elevation map, the various accumulations are further aligned to the same coordinate system on the Z-axis. Specifically:

[0100] Constructing an elevation map involves the following steps:

[0101] Step 1: Obtain the number of successful loop closures for each trajectory during the loop closure detection process;

[0102] During loop closure detection, the system identifies overlaps and similarities between trajectories to detect loops. Successful loop closures indicate that there is overlap or similarity between the two trajectories. The specific number of successful loop closures for each trajectory may vary; this step records the number of successful loop closures for each trajectory.

[0103] It should be noted that, in common techniques in this field, visual loop closure detection outputs a 6DOF pose. In this embodiment, since all trajectories are gravity-aligned and the purpose of this step is to address the alignment of the elevation coordinate system, the goal of loop closure detection in this embodiment is to find other target trajectories that overlap and match any given trajectory only in the XY space, while ignoring spatial matching in the Z space. Furthermore, the z component is obtained from the 6DOF pose, and the z component is used to calculate the elevation difference between trajectories with loops.

[0104] Step 2: Construct an elevation graph with the elevation value of each trajectory as a node, the elevation difference between any two trajectories with loops as an edge, and the number of successful loops corresponding to the elevation difference as the weight of the edge.

[0105] In this embodiment, the elevation map can also be called a Z-graph; specifically, the elevation value of each trajectory is used as a node. For any two trajectories with loops, if there are n successful loops between trajectory j and trajectory k, then... As an edge, its weight is set to n.

[0106] Furthermore, the maximum spanning tree algorithm is applied to the elevation map to obtain the maximum spanning tree model. It should be noted that the maximum spanning tree is a tree structure. In this embodiment, it is equivalent to a subgraph of the given elevation map, which contains all the trajectory nodes in the elevation map and has the maximum sum of the weights of all edges.

[0107] It should be noted that this embodiment specifically discusses how to use the maximum spanning tree to perform efficient and accurate alignment of the elevation map. The specific generation steps of the maximum spanning tree model can be achieved using the commonly used Kruskal algorithm and Prim algorithm, which will not be elaborated on in this embodiment.

[0108] Furthermore, after obtaining the maximum spanning tree model, it can be used to align the elevation values ​​of different trajectories. This step is achieved by propagating the elevation value of each node along the path of the maximum spanning tree, specifically including:

[0109] Step 1, Selecting the Root Node: First, select a node from the maximum spanning tree as the root node. In this embodiment, the node with the highest degree (i.e., the node with the most edges connected to it) is selected as the root node to facilitate the propagation of elevation values ​​in subsequent steps and reduce the computational load during the alignment process.

[0110] Step 2, Elevation value propagation: Starting from the root node, propagate the elevation value along the edges of the maximum spanning tree. For each edge, calculate the elevation difference between two adjacent nodes and apply this elevation difference to the elevation value of the child node. Continue to execute this step to gradually propagate the elevation value along the path of the maximum spanning tree until the leaf node is reached.

[0111] Step 3, Align Trajectory Elevation Values: This step utilizes the maximum spanning tree from the previous steps to propagate elevation values. For each trajectory, the elevation difference between each node and the root node can be accumulated by finding its corresponding node and following the path from the root node to that node. Finally, based on the elevation differences of each node relative to the root node, each trajectory is aligned in the Z-axis direction.

[0112] In this embodiment, a suitable reference trajectory can be found as the root node by constructing the maximum spanning tree. Furthermore, other trajectories can be propagated based on the root node (reference trajectory) to achieve altitude alignment. By aligning the altitude values ​​of all trajectories to the reference trajectory, a relatively consistent altitude information can be obtained.

[0113] Furthermore, when selecting a reference trajectory, it is only necessary to compare the features between trajectories that have intersections and loops, rather than comparing all trajectories. This can greatly reduce the number of comparisons and thus reduce the amount of computation.

[0114] Specifically, the process of optimizing the elevation map can be achieved using the following formula:

[0115]

[0116] Among them, the variables to be optimized are M elevation values ​​Z, N represents that there are a total of N edges among these elevation values ​​Z, i and j represent that an edge is composed of Zi and Zj. In addition, the first elevation Z1 will be fixed during optimization.

[0117] After the above steps, the alignment of each trajectory in the X, Y and Z directions has been completed. Furthermore, the aligned data can be used to achieve three-dimensional reconstruction. The three-dimensional reconstruction process can be achieved through image feature matching, triangulation and other means. Since the three-dimensional reconstruction steps are not related to the core inventive point of this application, they will not be described in detail in this embodiment.

[0118] Figure 3 This is a schematic diagram of the trajectory pose before elevation alignment according to the embodiments of this application. Figure 4 This is a schematic diagram of the trajectory pose after elevation alignment according to an embodiment of this application.

[0119] Through the above steps S101 to S103, compared with the traditional method of achieving 3D reconstruction based on constraints in the X, Y, and Z directions provided by RTK positioning information, this method utilizes a smart mobile terminal to run the SALM algorithm to collect gravity-aligned pitch and roll original mapping data. Furthermore, under the constraints of GPS information, 3DOF optimization is performed to further align the original mapping data in the X, Y directions and yaw, obtaining initial alignment data. Finally, based on the initial alignment data in the geographic coordinate system aligned in X, Y, pitch, roll, and yaw, an elevation map is constructed to align each trajectory in the Z-axis direction, and 3D reconstruction is performed using data aligned in the geographic coordinate system with all six degrees of freedom.

[0120] To address the issues of standard GPS information failing to provide usable altitude values ​​and varying altitudes across multiple tracks, this solution enables the use of a smart mobile terminal—a single handheld device—to collect data on surfaces with strong GPS signals and rapidly create maps at low cost.

[0121] The method provided in this embodiment allows ordinary personnel without professional skills to collect all the raw data required for mapping using a smartphone, and to align different trajectories using GPS information that does not have Z-axis direction constraints, thus enabling low-cost and rapid mapping of real-world scenes.

[0122] In some embodiments, after the above steps S101 to S103, aligned data with all six degrees of freedom aligned on the geographic coordinate system is obtained. However, due to the inherent absolute error of GPS positioning information and the cumulative relative error of each trajectory, the pose of the same image differs between different trajectories. This difference still affects the accuracy of the mapping results.

[0123] Considering the above problems, this embodiment optimizes the error between different trajectories by calculating a virtual reference frame. Specifically, it includes the following steps:

[0124] Step 1: Based on the poses of the images that have successfully looped through the loop in the visual loop closure detection process under different trajectories, determine the pose of the virtual reference frame. The orientation of the virtual reference frame pose (i.e., the rotation amount on Pitch, Roll, and Yaw) is the orientation of the image under its original trajectory after 3DOF optimization, and the position of the virtual reference frame pose (i.e., the translation amount on the X, Y, and Z axes) is the average of the pose translation amount of the image under the original trajectory and all successful loop closure trajectories.

[0125] Step 2: Under the premise of fixing the pose of the virtual reference frame, perform 6DOF optimization on the alignment data to obtain optimized alignment data; perform 3D reconstruction based on the optimized alignment data to obtain a 3D model.

[0126] It should be noted that in this embodiment, the observations including the geographic coordinate system only include the X and Y coordinates provided by the GPS positioning information, and the alignment gravity of the local coordinate system of the trajectory itself; that is, the Pitch and Roll are known and already aligned in the original data, while the yaw angle is data generated through secondary calculations based on GPS constraints.

[0127] The ZGraph elevation alignment module is optimized to calculate the least squares result of the relative elevation between multiple trajectories. The absolute elevation value is not observable in this application. The elevation value in this application is the relative height difference in the 6DOF relative pose obtained by image loop closure detection.

[0128] Since the calculation is based on relative elevation values, and when optimizing the 6DOF pose, providing only X, Y, and Z coordinates can further optimize the rotation part of the pose; therefore, this scheme only considers the X, Y, and Z coordinates when calculating the virtual reference frame pose, and performs simple vector averaging based on translation; there is no need to consider the rotation amount yaw, and the complexity of the calculation process is reduced accordingly.

[0129] After the above steps are completed, the reference frame will now calculate x_y_z. Since only displacement is involved and matrix operations between rotation angles are not required, the calculation of the virtual reference frame pose can be simplified to averaging the values ​​in the X, Y and Z directions. This can also ensure the correctness and validity of the reference frame pose calculation results.

[0130] In an alternative embodiment, taking image M of trajectory 1 as an example, assume it has been successfully registered to trajectory 2 and trajectory 3:

[0131] The pose of image M on trajectory 1 is T_M_to_Trajl, and its pose in the geographic coordinate system is T_M_to_Traj1;

[0132] The pose of image M on trajectory 2 is T_M_to_Traj2, and its pose in the geographic coordinate system is also T_M_to_Traj2;

[0133] The pose of image M on trajectory 3 is T_M_to_Traj3, and its pose in the geographic coordinate system is also T_M_to_Traj3;

[0134] Here, Geo (short for geography) represents the geographic coordinate system. Specifically, the calculation formula for the "virtual reference frame" is as follows:

[0135] Translation_M_to_Geo(mean)=(Translation_M_to_Geo(Traj_1)+Translation_M_to_Geo(Traj2)+……+Translation_M_to_Geo(Trajn)) / n;

[0136] Where Translation_M_to_Geo(Traj1), Translation_M_to_Geo(Traj2), and Translation_M_to_Geo(Trajn) are the positions of image M in the geographic coordinate system calculated under trajectory 1, trajectory 2, ..., trajectory n, respectively. Rotation_M_to_Geo(Trajk) is the orientation of image M in the geographic coordinate system under the 3DOF optimized trajectory K. Translation_M_to_Geo(mean) and Rotation_M_to_Geo(Trajk) constitute the pose of the virtual reference frame corresponding to image M in Trajjk.

[0137] It should be noted that after Zgraph alignment, the poses of all images in the trajectories are theoretically aligned to geographic coordinates in all six degrees of freedom. However, due to the inherent absolute error of GPS positioning information and the cumulative relative error of each trajectory, the poses of the same image M in trajectories 1, 2, and 3 may not be the same. In this embodiment, a reference image frame pose is obtained by averaging the poses of the same image under different trajectories. Although the pose of image M in the true geographic coordinate system cannot be known, the pose of the reference image frame provided in this embodiment can be used to approximately simulate and approach the true geographic coordinates of image M as closely as possible.

[0138] It is understandable that during the 6DOF optimization process, the pose of the virtual reference frame is set to fixed. Furthermore, after obtaining the mapping results, when visual localization is performed on image M again, the pose of image M under different trajectories is considered again, and the localization result is returned directly with the pose corresponding to the virtual reference frame. This is equivalent to the pose error of the same image M without providing a trajectory being eliminated, and the accuracy of the localization result is improved accordingly.

[0139] It's understandable that after the above steps, Zgraph only optimizes the z-axis, but the X, Y, Pitch, Roll, and Yaw axes are not optimized. Therefore, it's necessary to combine all degrees of freedom and perform another 6DOF optimization. Specifically, after obtaining the virtual reference frame, 6DOF optimization is achieved using the following formula:

[0140]

[0141] Wherein, K_inlier indicates that GPS noise has been removed, and Kinlier is less than or equal to K. w_k uses the weights obtained from Zgraph optimization. VL_ab represents the Visual Loop between Pose_a and Pose_b. In this embodiment, only EM noise estimation and removal are performed on VL. There are a total of N VL constraints. Each VL constraint represents the constraint between two Geo Pose a and b. Therefore, the subscripts in the formula are represented by a and b.

[0142] It is the constraint between the XY aligned pose in geographic coordinates and the relative pose in the trajectory acquired by the SLAM algorithm, || GPS k -Geo k || 2 It refers to the constraints between the GPS positioning information and the pose (Geo) to be optimized. These are the constraints between the poses to be optimized (Geo_a and Geo_b) and the visual loop. It is used to eliminate erroneous GPS constraints and updates w using the Expectation-Maximization algorithm. k Additionally, similar to the 3DOF formula above, the "-" minus sign involved in the Pose calculation in this step represents calculating the difference between two Pose matrices, where the inverse of the Pose matrix to the left of the minus sign is multiplied by the Pose matrix to the right of the minus sign.

[0143] Optional, Figure 5 This is a schematic diagram of a trajectory pose optimized with 6DOF according to another embodiment of this application, as shown below. Figure 5 As shown, after 6DOF optimization with a fixed virtual reference frame, the elevations of each trajectory are perfectly aligned.

[0144] Through the above steps, after obtaining the pose of the virtual reference frame, the virtual reference frame is fixed, and the initial alignment data is optimized using 6DOF. This aims to make the pose of the same image across different trajectories as close as possible to the true value, thus minimizing errors. By using the 6DOF-optimized pose for triangulation and subsequent mapping, the accuracy of the mapping results can be further improved.

[0145] Figure 6 This is a schematic diagram illustrating one method of obtaining a virtual reference frame according to an embodiment of this application. Figure 7 This is a schematic diagram illustrating 6DOF optimization under the premise of a fixed virtual reference frame pose, according to an embodiment of this application.

[0146] This embodiment also provides a low-cost 3D mapping system based on a smart handheld mobile terminal. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0147] Figure 8 This is a structural block diagram of a low-cost 3D mapping system based on a smart handheld mobile terminal according to an embodiment of this application. The system includes: an acquisition module 80, an alignment module 81, and a reconstruction module 82, wherein:

[0148] The acquisition module 80 is used to acquire initial mapping data collected by a smart handheld mobile terminal. The initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes an image and gravity-aligned pose and GPS positioning information corresponding to the image.

[0149] The optimization module uses 81 to perform 3DOF optimization on the initial mapping data with GPS positioning information as a constraint, aligning the horizontal direction and heading angle of each trajectory to obtain the initial alignment data in X, Y, pitch, roll and yaw relative to the geographic coordinate system.

[0150] Furthermore, an elevation map is constructed based on the initial alignment data. The elevation map is optimized by the maximum spanning tree, aligning the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, resulting in aligned data in which all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data is a relative value.

[0151] The reconstruction module 82 is used to reconstruct a 3D scene based on the alignment data.

[0152] Compared to traditional methods that use RTK positioning information to provide constraints in the X, Y, and Z directions for 3D reconstruction, this application utilizes a smart mobile terminal, a civilian device, to collect data. Based on this data, an elevation map is constructed to align the trajectories along the Z-axis. Finally, the aligned data is used for 3D reconstruction. Users can use a smartphone, a handheld device, to collect data on the ground with good GPS signal to achieve low-cost and rapid 3D mapping.

[0153] also, Figure 9 This is a schematic diagram of another low-cost 3D mapping system based on a smart handheld mobile terminal according to an embodiment of this application.

[0154] In one embodiment, Figure 10 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 10 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 10 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system and computer programs to run, the computer programs are executed by the processor to implement various low-cost 3D mapping methods based on smart handheld mobile terminals, and the database stores data.

[0155] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0156] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0157] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A low-cost 3D mapping method based on a smart handheld mobile terminal, characterized in that, The method includes: Acquire initial mapping data collected through a smart handheld mobile terminal, wherein the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes an image and gravity-aligned pose and GPS positioning information corresponding to the image; Using the GPS positioning information as constraint information, the initial mapping data is optimized using 3DOF, and the horizontal direction and heading angle of each trajectory are aligned to obtain the initial alignment data that is aligned with the X, Y, pitch, roll and yaw relative to the geographic coordinate system. An elevation map is constructed based on the initial alignment data. The elevation map is then optimized using a maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, resulting in aligned data where all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data is a relative value. Constructing an elevation map based on the initial alignment data includes: Perform visual loop closure detection on each trajectory and obtain the number of successful loop closures for each trajectory in the loop closure detection. The elevation graph is constructed by using the elevation values ​​of each trajectory as nodes, the elevation difference between any two trajectories with loops as edges, and the number of successful loops corresponding to the elevation difference as the weight of the edges. Select a node of the maximum spanning tree as the root node, obtain the elevation difference between each node and the root node, and align each trajectory in the Z-axis direction according to the elevation difference of each node relative to the root node. Three-dimensional scene reconstruction is performed based on the alignment data.

2. The method according to claim 1, characterized in that, The method further includes: For each trajectory in the initial alignment data, feature point matching and triangulation of the image are performed to obtain the three-dimensional point cloud information of each trajectory in the XY space; Based on the three-dimensional point cloud information, visual loop closure detection is performed on each trajectory to obtain the relative pose transformation of each trajectory relative to other target trajectories that overlap and match with it in XY space. Based on the relative pose transformation, the altitude difference between each trajectory and the other target trajectories is obtained.

3. The method according to claim 1, characterized in that, The elevation map is optimized using a maximum spanning tree, aligning the elevation values ​​of each trajectory in the initial aligned data to obtain aligned data including: The maximum spanning tree algorithm is applied to the elevation map to obtain the maximum spanning tree model corresponding to the elevation values ​​between nodes; The root node of the maximum spanning tree model is determined, and the elevation value is propagated based on the maximum spanning tree model to calculate the elevation difference between other nodes and the root node, wherein the elevation value of the root node is set to 0. Based on the elevation difference between each node and the root node, the elevation values ​​of all nodes are aligned to obtain aligned data in the geographic coordinate system, where all six degrees of freedom are aligned.

4. The method according to claim 1, characterized in that, Aligning the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, and obtaining the alignment data, aligning multiple trajectories to the geographic coordinate system in six degrees of freedom (X, Y, Z, Pitch, Roll, and Yaw), the method further includes: Based on the poses of the successfully looped images under different trajectories during the visual loop closure detection process, the pose of the virtual reference frame is determined. The orientation of the virtual reference frame pose is the orientation of the image under its original trajectory after 3DOF optimization, and the position of the virtual reference frame pose is the average of the pose translation of the image under the original trajectory and all successfully looped trajectories. Under the premise of fixing the pose of the virtual reference frame, the alignment data is optimized by 6DOF to obtain the optimized alignment data; Based on the optimized aligned data, a 3D reconstruction is performed to obtain a 3D model.

5. The method according to claim 4, characterized in that, Determining the pose of a virtual reference frame based on the pose of any image across multiple trajectories includes: The pose of the virtual reference frame corresponding to image M in Trajk is calculated using the following formula; Translation_M_to_Geo(mean)=(Translation_M_to_Geo(Traj1)+Translation_M_to_Geo(Traj2)+ ……+Translation_M_to_Geo(Trajn)) / n; Wherein, Translation_M_to_Geo(Traj1), Translation_M_to_Geo(Traj2), and Translation_M_to_Geo(Trajn) are the positions of image M in the geographic coordinate system calculated under trajectory 1, trajectory 2, ..., trajectory n, respectively, and Rotation_M_to_Geo(Trajk) is the orientation of image M in the geographic coordinate system under the 3DOF optimized trajectory K. Translation_M_to_Geo(mean) and Rotation_M_to_Geo(Trajk) constitute the pose of the virtual reference frame corresponding to image M in Trajk.

6. A low-cost 3D mapping system based on a smart handheld mobile terminal, characterized in that, The system includes: an acquisition module, an alignment module, and a reconstruction module, wherein: The acquisition module is used to acquire initial mapping data collected by a smart handheld mobile terminal. The initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes an image and gravity-aligned pose and GPS positioning information corresponding to the image. The optimization module is used to perform 3DOF optimization on the initial mapping data with the GPS positioning information as a constraint, align the horizontal direction and heading angle of each trajectory, and obtain the initial alignment data that is aligned in X, Y, pitch, roll and yaw relative to the geographic coordinate system. Furthermore, an elevation map is constructed based on the initial alignment data, and the elevation map is optimized by the maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, thereby obtaining aligned data in which all six degrees of freedom are aligned on the geographic coordinate system, wherein the Z-axis data in the aligned data is a relative value; The reconstruction module is used to reconstruct a three-dimensional scene based on the alignment data; The construction of the elevation map based on the initial alignment data includes: performing visual loop closure detection on each trajectory and obtaining the number of successful loop closures for each trajectory in the loop closure detection. The elevation graph is constructed by using the elevation values ​​of each trajectory as nodes, the elevation difference between any two trajectories with loops as edges, and the number of successful loops corresponding to the elevation difference as the weight of the edges. Select a node of the maximum spanning tree as the root node, obtain the elevation difference between each node and the root node, and align each trajectory in the Z-axis direction based on the elevation difference of each node relative to the root node.

7. The system according to claim 6, characterized in that, The system also includes an image algorithm module, wherein, The image algorithm module is used for: For each trajectory in the initial alignment data, feature point matching and triangulation of the image are performed to obtain the three-dimensional point cloud information of each trajectory in the XY space; Based on the three-dimensional point cloud information, visual loop closure detection is performed on each trajectory to obtain the relative pose transformation of each trajectory relative to other target trajectories that overlap and match with it in XY space. Based on the relative pose transformation, the altitude difference between each trajectory and the other target trajectories is obtained.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images