A low-cost visual positioning method and system based on unstructured maps

Abstract

The application relates to a low-cost visual positioning method, system, computer device and computer readable storage medium based on a structureless map, which is based on a structureless map without 3D points, does not need PNP calculation, and can obtain a visual positioning pose through 2D image matching and relative pose estimation based on a deep learning algorithm. Since there is no 3D structure in the visual positioning map, the memory occupation of the server can be greatly reduced, and the updating and expansion of the map are also more simple and efficient. Meanwhile, since only the image needs to be encrypted and the 3D point does not need to be encrypted, the privacy protection feature of the map data is more easily realized. In addition, the visual positioning method provided by the application has significant improvement in positioning efficiency and success rate.

Description

Technical Field

[0001] This application relates to the field of visual positioning, and in particular to a low-cost visual positioning method, system, computer device, and computer-readable storage medium based on unstructured maps. Background Technology

[0002] SFM (Structure from Motion) is a key technology and method that can recover the structure of a 3D scene from a series of 2D images. In this technology, building a visual map requires simultaneously recovering the 3D structure and camera pose, which has high computational complexity.

[0003] Conventional VPS (Visual Positioning System) technology retrieves the N most similar images, matches the most similar 2D pixel feature points, and further calculates the 6DOF (degree of freedom) pose using PnP (Project n Points) technology (calculating image pose through 3D-2D matching pairs). This conventional VPS relies heavily on the 3D structure of the map, resulting in high computational complexity in the visual positioning process. Furthermore, map additions and updates require merging the 3D structure, leading to high map maintenance costs.

[0004] Therefore, existing technologies have not yet solved the problem of excessively high computational and operational costs for visual mapping and visual positioning. Summary of the Invention

[0005] This application provides a low-cost visual positioning method, system, computer device, and computer-readable storage medium based on unstructured maps, to at least solve the problem of excessively high computational and operational costs for visual positioning in related technologies.

[0006] In a first aspect, embodiments of this application provide a low-cost visual localization method based on unstructured maps, characterized in that the method includes:

[0007] Based on the initial mapping data aligned by gravity, an unstructured map is constructed with the pose of each image aligned to the geographic coordinate system. The initial mapping data includes multiple trajectories, each trajectory includes multiple sets of tuple data, and each tuple data includes an image and image pose and GPS positioning information.

[0008] Receive the image to be located, and determine candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map.

[0009] The relative poses between the image to be located and each candidate map frame are obtained using a deep learning algorithm. Based on the relative poses and the poses of the candidate map frames aligned to the geographic coordinate system in the unstructured map, the absolute pose corresponding to the image to be located is obtained.

[0010] In some embodiments, determining candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map includes:

[0011] The image to be located is matched one by one with all the map frames in the unstructured map using a deep learning matching algorithm, and the candidate map frames are determined based on the similarity obtained from the matching.

[0012] Alternatively, obtain the GPS information corresponding to the image to be located, determine the target area in the unstructured map based on the GPS information, and select the first candidate frame in the target area.

[0013] The similarity between the image to be located and each of the first candidate frames is obtained by using a deep learning matching algorithm, and the candidate map frame is determined in the first candidate frames based on the similarity.

[0014] In some embodiments, after obtaining the absolute pose corresponding to the image to be located, the method further includes:

[0015] For each absolute pose, calculate the pose difference between it and other absolute poses;

[0016] Determine whether the pose differences corresponding to each absolute pose are less than a preset error threshold. If so, add a voting mark to the absolute pose based on the number of pose differences that are less than the preset error threshold.

[0017] The absolute pose with the most votes is recorded as the optimal absolute pose, and the associated poses that have a voting relationship with the optimal absolute pose are obtained.

[0018] An EM pose graph is constructed based on the optimal absolute pose and the associated pose. The optimized absolute pose is obtained by optimizing the EM pose graph.

[0019] Based on the optimized absolute pose, the visual localization result corresponding to the image to be localized is returned.

[0020] In some embodiments, the method further includes:

[0021] If multiple absolute poses have the same voting label, determine whether the rotation errors of these multiple absolute poses are inconsistent. If so, select the absolute pose with the smallest rotation error as the optimal absolute pose.

[0022] If not, determine whether the translation errors of the multiple absolute poses are inconsistent. If so, select the absolute pose with the smallest translation error as the optimal absolute pose.

[0023] In some embodiments, constructing an unstructured map with the poses of each image aligned to a geographic coordinate system based on initial gravity-aligned mapping data includes:

[0024] Using the GPS positioning information as constraint information, the initial mapping data is optimized using 3DOF, and the horizontal direction and heading angle of each trajectory are aligned to obtain the initial alignment data that is aligned with the X, Y, pitch, roll and yaw relative to the geographic coordinate system.

[0025] An elevation map is constructed based on the initial alignment data. The elevation map is then optimized using a maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, resulting in aligned data where all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data is a relative value.

[0026] The unstructured map is constructed based on the alignment data, wherein the unstructured map includes a series of images and each image aligned to a six-degree-of-freedom pose in geographic coordinates.

[0027] In some embodiments, the elevation map is optimized using a maximum spanning tree to align the elevation values ​​of each trajectory in the initial alignment data, resulting in aligned data including:

[0028] The maximum spanning tree algorithm is applied to the elevation map to obtain the maximum spanning tree model corresponding to the elevation values ​​between nodes;

[0029] The root node of the maximum spanning tree model is determined, and the elevation value is propagated based on the maximum spanning tree model to calculate the elevation difference between other nodes and the root node, wherein the elevation value of the root node is set to 0.

[0030] Based on the elevation difference between each node and the root node, the elevation values ​​of all nodes are aligned to obtain aligned data in the geographic coordinate system, where all six degrees of freedom are aligned.

[0031] In some embodiments, the elevation values ​​of each trajectory in the initial alignment data are aligned to the same coordinate system. After obtaining the alignment data, multiple trajectories are aligned to the geographic coordinate system in six degrees of freedom: X, Y, Z, Pitch, Roll, and Yaw. The method further includes:

[0032] Based on the poses of the successfully looped images under different trajectories during the visual loop closure detection process, the pose of the virtual reference frame is determined. The orientation of the virtual reference frame pose is the orientation of the image under its original trajectory after 3DOF optimization, and the position of the virtual reference frame pose is the average of the pose translation of the image under the original trajectory and all successfully looped trajectories.

[0033] With the virtual reference frame pose fixed, the alignment data is optimized using 6DOF to obtain the optimized alignment data.

[0034] The unstructured map is constructed based on the optimized aligned data.

[0035] Secondly, embodiments of this application provide a low-cost visual positioning system based on an unstructured map, the system comprising: a map building module, a search module, and a positioning module, wherein:

[0036] The map building module is used to construct an unstructured map with the poses of each image aligned to the geographic coordinate system based on the initial mapping data with gravity alignment. The initial mapping data includes multiple trajectories, each of which includes an image and image pose and GPS positioning information.

[0037] The search module is used to receive an image to be located and determine candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map.

[0038] The localization module is used to obtain the relative pose between the image to be localized and each candidate map frame through a deep learning algorithm, and to obtain a visual localization result based on the relative pose and the pose of the candidate map frame aligned to the geographic coordinate system in the unstructured map.

[0039] Thirdly, embodiments of this application provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0041] Compared to related technologies, this application provides a low-cost visual positioning method based on an unstructured map. Using an unstructured map that does not include 3D points, it eliminates the need for PNP calculations and obtains the visual positioning pose through 2D image matching and relative pose estimation based on deep learning algorithms. Since the visual positioning map lacks 3D structure, server memory usage is significantly reduced, and map updates and expansions are simpler and more efficient. Furthermore, because only the image needs to be encrypted, without encrypting 3D points, map data privacy protection is easier to achieve. In addition, the visual positioning method provided in this application significantly improves positioning efficiency and success rate. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0043] Figure 1 This is a flowchart of a low-cost visual localization method based on an unstructured map according to an embodiment of this application;

[0044] Figure 2 This is a flowchart of constructing an unstructured map according to an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of a 3DOF optimization according to an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the trajectory pose before elevation alignment according to the embodiments of this application;

[0047] Figure 5 This is a schematic diagram of the trajectory pose after elevation alignment according to an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of a trajectory pose optimized by 6DOF according to another embodiment of this application;

[0049] Figure 7 This is a schematic diagram illustrating one method of obtaining a virtual reference frame according to an embodiment of this application;

[0050] Figure 8 This is a schematic diagram illustrating 6DOF optimization under the premise of a fixed virtual reference frame pose, according to an embodiment of this application.

[0051] Figure 9 This is a structural block diagram of a low-cost visual positioning system based on an unstructured map according to an embodiment of this application;

[0052] Figure 10This is a schematic diagram of another low-cost visual positioning system based on an unstructured map according to an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0055] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0058] Visual positioning is a commonly used positioning technology in the AR field. Typically, images are captured using devices such as mobile phones in a large space (such as a museum or exhibition hall) to be experienced. After constructing a visual map using SFM (Structure From Motion), a VPS (Visual Positioning Service) is provided, allowing users to experience the AR content placed there.

[0059] SFM (Search Engine Optimization) builds visual maps by simultaneously recovering the 3D structure and camera pose, resulting in high computational complexity. Conventional visual positioning techniques retrieve the N most similar images from a 3D map, then match the most similar 2D pixel feature points, and subsequently perform Project n Points (PnP, a technique for calculating image pose through 3D-2D matching pairs) to calculate the 6DOF (degrees of freedom) pose. Therefore, conventional VPSs heavily rely on the map's 3D structure. However, 3D structure maps built using the SFM method are cumbersome to supplement and update, requiring the merging of 3D structures, leading to high map maintenance costs. Furthermore, since each positioning requires PnP (Project n Points) through 3D-2D matching pairs, this type of positioning method is highly complex and inefficient.

[0060] In view of this, embodiments of this application provide a low-cost visual localization method based on unstructured maps. Figure 1 This is a flowchart of a low-cost visual localization method based on an unstructured map according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0061] S101, Based on the initial mapping data aligned by gravity, construct an unstructured map with the pose of each image aligned to the geographic coordinate system. The initial mapping data includes multiple trajectories, each of which includes an image and image pose and GPS positioning information.

[0062] Specifically, Figure 2 This is a flowchart of constructing an unstructured map according to an embodiment of this application, such as... Figure 2 As shown, constructing an unstructured map involves the following steps:

[0063] S1011, acquire initial mapping data collected by a smart handheld mobile terminal, wherein the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes the gravity alignment pose and GPS positioning information corresponding to the image.

[0064] Personnel can use smartphones to collect the initial mapping data in a low-cost, efficient and convenient way by deploying and running SLAM algorithms. Specifically, the program or application that supports running SLAM algorithms can be ARKit on iOS or ARCore on Android.

[0065] Furthermore, the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes the gravity-aligned pose and GPS positioning information corresponding to the image; in this embodiment, the trajectory is composed of a set of continuous images and the relative pose of each image in local coordinates.

[0066] Furthermore, the pose acquired by the SLAM algorithm is itself gravity-aligned, which means rotating the Z-axis in the SLAM world coordinate system to align with the direction of gravity. This implies that the object's roll and pitch angles have been corrected relative to the direction of gravity, meaning that pitch and roll are aligned, but yaw may not be aligned.

[0067] S1012, using GPS positioning information as constraint information, performs 3DOF optimization on the initial mapping data, aligns the horizontal direction and heading angle of each trajectory, and obtains the initial alignment data in X, Y, pitch, roll and yaw relative to the geographic coordinate system.

[0068] It is understandable that the data obtained in step S1011 are in their respective local coordinate systems. Although they are gravity aligned (i.e., pitch and roll aligned), their XY and yaw are not aligned, and their elevations are not aligned either.

[0069] Figure 3 This is a schematic diagram of a 3DOF optimization according to an embodiment of this application, such as... Figure 3 As shown, in this step, the XY plane of multiple trajectories is optimized by the 3DOF optimization module so that each trajectory is aligned with the geographic coordinate system in the horizontal direction.

[0070] Specifically, the 3DOF optimization process can align each trajectory to the geographic coordinate system horizontally using the following formula:

[0071]

[0072] Where Geo is the variable to be optimized, i.e., the pose of the image aligned to the geographic coordinate system; GPS represents the constraints provided by GPS positioning information; Track is the pose in the local coordinate system; and w k These are weighting coefficients used to eliminate incorrect GPS constraints; the minus sign "-" in the formula for calculating Pose means calculating the difference between two Pose matrices, where the inverse of the Pose matrix to the left of the minus sign is multiplied by the Pose matrix to the right of the minus sign.

[0073] It is the constraint between the XY aligned pose in geographic coordinates and the relative pose in the trajectory acquired by the SLAM algorithm, || GPS k -Geok|| 2 It refers to the constraint between GPS positioning information and the pose to be optimized. It is used to eliminate erroneous GPS constraints and updates w using the Expectation-Maximization algorithm. k .

[0074] In summary, this optimization formula takes into account the relative constraints between trajectories, the constraints between GPS positioning information and the actual geographic pose. The goal of this formula is to achieve 3DoF optimization by minimizing the error under constraints, and to align different trajectories on the X-axis, Y-axis and yaw.

[0075] S1013, construct an elevation map based on the initial alignment data, optimize the elevation map by using the maximum spanning tree, align the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, and obtain aligned data in which all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data is a relative value. Based on the aligned data, construct an unstructured map.

[0076] It should be noted that some preprocessing steps need to be performed before this step to provide basic information for constructing the elevation map. Specifically, these steps include the following:

[0077] Step 1: For each trajectory in the initial alignment data, perform feature point matching and triangulation of the image to obtain the three-dimensional point cloud information of each trajectory in the XY space.

[0078] Step 2: Based on the 3D point cloud information, perform visual loop closure detection on each trajectory to obtain the relative pose transformation of each trajectory relative to other target trajectories that overlap and match with it in the XY space.

[0079] Step 3: Based on the relative pose transformation, obtain the altitude difference between each trajectory and other target trajectories.

[0080] Furthermore, since the raw data collected in S1011 is already gravity-aligned, i.e., Pitch and Roll are already aligned, after step S1012, which performs 3DoF optimization under GPS constraints, each trajectory is aligned in the X and Y directions and in the Yaw direction. At this point, each trajectory in the raw data is already aligned in the five degrees of freedom: X, Y, Pitch, Roll, and Yaw. In this step, by constructing an elevation map, each trajectory is further aligned to the same coordinate system on the Z-axis.

[0081] Specifically, constructing an elevation map includes the following steps:

[0082] Step 1: Obtain the number of successful loop closures for each trajectory during the loop closure detection process;

[0083] During loop closure detection, the system identifies overlaps and similarities between trajectories to detect loops. Successful loop closures indicate that there is overlap or similarity between the two trajectories. The specific number of successful loop closures for each trajectory may vary; this step records the number of successful loop closures for each trajectory.

[0084] It should be noted that, in common techniques in this field, visual loop closure detection outputs a 6DOF pose. In this embodiment, since all trajectories are gravity-aligned and the purpose of this step is to address the alignment of the elevation coordinate system, the goal of loop closure detection in this embodiment is to find other target trajectories that overlap and match any given trajectory only in the XY space, while ignoring spatial matching in the Z space. Furthermore, the z component is obtained from the 6DOF pose, and the z component is used to calculate the elevation difference between trajectories with loops.

[0085] Step 2: Construct an elevation graph with the elevation value of each trajectory as a node, the elevation difference between any two trajectories with loops as an edge, and the number of successful loops corresponding to the elevation difference as the weight of the edge.

[0086] In this embodiment, the elevation map can also be called a Z-graph; specifically, the elevation value of each trajectory is used as a node. For any two trajectories with loops, if there are n successful loops between trajectory j and trajectory k, then... As an edge, its weight is set to n.

[0087] Furthermore, the maximum spanning tree algorithm is applied to the elevation map to obtain the maximum spanning tree model. It should be noted that the maximum spanning tree is a tree structure. In this embodiment, it is equivalent to a subgraph of the given elevation map, which contains all the trajectory nodes in the elevation map and has the maximum sum of the weights of all edges.

[0088] After obtaining the maximum spanning tree model, it can be used to align the elevation values ​​of different trajectories. This step is achieved by propagating the elevation value of each node along the path of the maximum spanning tree, specifically including:

[0089] Step 1, Select the root node: First, select a node from the maximum spanning tree as the root node. In this embodiment, the node with the highest degree (i.e., the node with the most edges connected to it) is selected as the root node to facilitate the propagation of elevation values ​​in subsequent steps and reduce the amount of computation in the alignment process;

[0090] Step 2, Elevation value propagation: Starting from the root node, propagate the elevation value along the edges of the maximum spanning tree. For each edge, calculate the elevation difference between two adjacent nodes and apply this difference to the elevation value of the child node. Continue to execute this step to gradually propagate the elevation value along the path of the maximum spanning tree until the leaf node is reached.

[0091] Step 3, Align Trajectory Elevation Values: Utilize the maximum spanning tree from the previous steps to propagate elevation values. For each trajectory, the elevation difference between each node and the root node can be accumulated by finding its corresponding node and following the path from the root node to that node. Finally, based on the elevation differences of each node relative to the root node, align each trajectory in the Z-axis direction.

[0092] In this embodiment, by constructing a maximum spanning tree, a suitable reference trajectory can be found as the root node. Furthermore, other trajectories can be propagated based on the root node (reference trajectory) to achieve altitude alignment. By aligning the altitude values ​​of all trajectories to the reference trajectory, a relatively consistent altitude information can be obtained.

[0093] Furthermore, when selecting a reference trajectory, it is only necessary to compare the features between trajectories that have intersections and loops, rather than comparing all trajectories. This can greatly reduce the number of comparisons and thus reduce the amount of computation.

[0094] Specifically, the process of optimizing the elevation map can be achieved using the following formula:

[0095]

[0096] Among them, the variables to be optimized are M elevation values ​​Z, N represents that there are a total of N edges among these elevation values ​​Z, i and j represent that an edge is composed of Zi and Zj. In addition, the first elevation Z1 will be fixed during optimization.

[0097] Figure 4 This is a schematic diagram of the trajectory pose before elevation alignment according to the embodiments of this application. Figure 5 This is a schematic diagram of the trajectory pose after elevation alignment according to an embodiment of this application.

[0098] In an optional embodiment, after obtaining the aligned data in which all six degrees of freedom are aligned on the geographic coordinate system, the pose of the same image differs between different trajectories due to the inherent absolute error of GPS positioning information and the cumulative relative error of each trajectory. This difference will still affect the accuracy of the mapping results.

[0099] Considering the above problems, this embodiment optimizes the error between different trajectories by calculating a virtual reference frame. Specifically, it includes the following steps:

[0100] Step 1: Based on the poses of the images that have successfully looped through the loop in the visual loop closure detection process under different trajectories, determine the pose of the virtual reference frame. The orientation of the virtual reference frame pose (i.e., the rotation amount on Pitch, Roll, and Yaw) is the orientation of the image under its original trajectory after 3DOF optimization, and the position of the virtual reference frame pose (i.e., the translation amount on the X, Y, and Z axes) is the average of the pose translation amount of the image under the original trajectory and all successful loop closure trajectories.

[0101] Step 2: Under the premise of fixing the pose of the virtual reference frame, perform 6DOF optimization on the alignment data to obtain the optimized alignment data; obtain the unstructured map based on the optimized alignment data; Figure 6 This is a schematic diagram of a trajectory pose optimized by 6DOF according to another embodiment of this application.

[0102] It should be noted that in this embodiment, the observations including the geographic coordinate system only include the X and Y coordinates provided by the GPS positioning information, and the alignment gravity of the local coordinate system of the trajectory itself; that is, the Pitch and Roll are known and already aligned in the original data, while the yaw angle is data generated through secondary calculations based on GPS constraints.

[0103] The ZGraph elevation alignment module is optimized to calculate the least squares result of the relative elevation between multiple trajectories. The absolute elevation value is not observable in this application. The elevation value in this application is the relative height difference in the 6DOF relative pose obtained by image loop closure detection.

[0104] Since the calculation is based on relative elevation values, and when optimizing the 6DOF pose, providing only X, Y, and Z coordinates can further optimize the rotation part of the pose; therefore, this scheme only considers the X, Y, and Z coordinates when calculating the virtual reference frame pose, and performs simple vector averaging based on translation; there is no need to consider the rotation amount yaw, and the complexity of the calculation process is reduced accordingly.

[0105] After the above steps are completed, the reference frame will now calculate xyz. Since only displacement is involved and matrix operations between rotation angles are not required, the calculation of the virtual reference frame pose can be simplified to averaging the values ​​in the X, Y and Z directions. This can also ensure the correctness and validity of the reference frame pose calculation results.

[0106] After the above steps of pose alignment and pose optimization based on virtual reference frames, we have obtained aligned data with a certain accuracy in six degrees of freedom. It can be understood that the obtained aligned data includes a series of images and their corresponding poses in geographic coordinates. The unstructured map constructed based on this aligned data includes multiple image frames, and each image frame has a geometric label, which is the 6DOF pose of the image aligned in geographic coordinates.

[0107] Figure 7 This is a schematic diagram illustrating one method of obtaining a virtual reference frame according to an embodiment of this application. Figure 8 This is a schematic diagram illustrating 6DOF optimization under the premise of a fixed virtual reference frame pose, according to an embodiment of this application.

[0108] Through steps S1011 to S1013, the SALM algorithm is used on a smart mobile terminal to collect gravity-aligned pitch and roll raw mapping data. Further, under the constraint of GPS information, 3DOF optimization is performed to further align the raw mapping data in the X and Y directions and the yaw direction, obtaining initial alignment data. Finally, based on the initial alignment data aligned in the geographic coordinate system in X, Y, pitch, roll, and yaw, an elevation map is constructed to align each trajectory in the Z-axis direction. Using data aligned in the geographic coordinate system across all six degrees of freedom, an unstructured map is obtained. This map does not require matching operations between 2D and 3D points, has a lower computational load, and does not contain 3D structure.

[0109] S102, Receive the image to be located, and determine candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map.

[0110] It is understandable that the image to be located should also be an image of the offline scene corresponding to the aforementioned unstructured map. Furthermore, the candidate map frame is determined based on the image to be located in the following two cases:

[0111] 1) In indoor spaces such as subway stations and exhibition halls;

[0112] Since GPS information in indoor spaces is usually less accurate or GPS signals are not available, a deep learning matching algorithm can be used to extract global image descriptors from the image to be located, and then match the global descriptors with the global descriptors of all map frames in the unstructured map one by one. Candidate map frames are determined based on the similarity obtained from the matching.

[0113] 2) In open outdoor settings such as scenic spots and parks;

[0114] In such scenarios, since GPS signals have good accuracy, it is optional to obtain the GPS information corresponding to the image to be located (i.e., the location information collected by the mobile phone when the image is taken).

[0115] First, the target area is determined in the unstructured map based on GPS information, which is equivalent to performing a coarse screening to narrow down the range of subsequent image matching. Furthermore, the first candidate frame is selected within this range for subsequent image similarity matching, which can also reduce the amount of computation and improve computational efficiency. Optionally, all images within a 12m radius of the unstructured map can be searched based on the GPS information of the image to be located to obtain the aforementioned first candidate frame.

[0116] Furthermore, a deep learning matching algorithm is used to obtain the similarity between the image to be located and each first candidate frame, and candidate map frames are determined from the first candidate frames based on the similarity.

[0117] Specifically, a certain threshold can be set to search for up to N image candidate frames in the image candidate frames. The image candidate frames must satisfy the condition that the similarity score with the image to be located is greater than the threshold. If no image is found, the localization fails.

[0118] It should be noted that the deep learning algorithm used in step S102 for extracting global image descriptors and performing similarity matching is preferably the MixVPR algorithm. The MixVPR algorithm can capture key information in the image more effectively through an innovative feature fusion method.

[0119] In addition, other deep learning algorithms such as HFNet can be used to extract global image descriptors and perform similarity matching.

[0120] S103: Using a deep learning algorithm, the relative pose between the image to be located and each candidate map frame is obtained. Based on the relative pose and the pose of the candidate map frames aligned to the geographic coordinate system in the unstructured map, the absolute pose corresponding to the image to be located is obtained.

[0121] In this embodiment, this step can estimate the above-mentioned phase pose using an algorithm based on the GRelPose (Generalizable End-to-EndRelative Camera Pose Regression) deep learning framework;

[0122] In an exemplary embodiment, obtaining the relative pose includes the following steps:

[0123] Step 1, Input Image: The system accepts a localization image I and N candidate frames {F1, F2, ..., FN};

[0124] Step 2, Feature Extraction: Use a deep convolutional network to extract high-dimensional feature vectors vI and vFi for each image I and Fi;

[0125] Step 3, Feature Matching and Fusion: Match and fuse the features of the localized image and the candidate frame to generate fused features vI, Fi;

[0126] Step 4, Pose Regression: Input the fused features vI, Fi into the regression network to predict the relative pose pi, where pi = (Ri, ti), R_iRi is the rotation matrix, and t_iti is the translation vector;

[0127] Step 5, Output Results: The final output is the N relative poses {p1, p2, ..., pN} between the localized image and each candidate frame.

[0128] It is understandable that by using the deep learning method of GrelPose through the above steps, high-precision relative pose estimation can be provided. The pose estimation process, from feature extraction to pose regression, is trained end-to-end, which reduces error accumulation. Moreover, the algorithm has good generalization ability and can adapt to different scenarios and datasets.

[0129] It should be noted that the algorithm based on the GrelPose deep learning framework is only one possible method for estimating relative pose in this application. Alternatively, the following methods can also be used to estimate relative pose:

[0130] 1)Map-free Visual Relocalization: Metric Pose Relative to a SingleImage、

[0131] 2)Visual Camera Re-LocalizationUsing Graph Neural Networks andRelative Pose Supervision

[0132] 3)Insights on Evaluation of Camera Re-localizationUsing Relative PoseRegression

[0133] Furthermore, after obtaining the relative pose T_loc_to_camera_n between the image to be located and each candidate map frame, the absolute pose T_camera_n_to_world between the image to be located and the candidate map frame aligned to geographic coordinates in the unstructured map can be used to calculate the absolute pose between the image to be located and the map, i.e., T_loc(n)_to_world = T_loc_to_camera_n * T_camera_n_to_world;

[0134] Furthermore, considering that there may be erroneous frames in the candidate map frames obtained based on similarity matching, the absolute pose obtained directly from the above steps may not be very accurate. In this embodiment, after the above steps, the absolute pose is further optimized by constructing an EM pose graph, and the optimized pose is returned to the terminal device that sent the positioning request. Specifically, this includes the following steps:

[0135] S1031, Filter out abnormal poses from multiple poses;

[0136] In this embodiment, multiple absolute poses are obtained based on multiple candidate map frames through the above steps. A certain threshold and voting mechanism can be set to filter out poses that are obviously abnormal. Specifically, the steps include the following:

[0137] Step 1: For each absolute pose, calculate the pose difference between it and other absolute poses, and determine whether each pose difference corresponding to each absolute pose is less than a preset error threshold. If so, add a voting mark to the absolute pose based on the number of pose differences that are less than the preset error threshold.

[0138] Step 2: Record the absolute pose with the most votes as the optimal absolute pose, and obtain the associated poses that are associated with the optimal absolute pose through voting.

[0139] If there are multiple absolute poses with the same number of votes, the pose with the smallest rotation error is selected as the optimal absolute pose; furthermore, if there are multiple absolute poses with the same number of votes and the same rotation error, the pose with the smallest translation error is selected as the optimal absolute pose.

[0140] A specific example of this process can be described as follows:

[0141] 1. Calculate the pose difference and vote:

[0142] 1.1 For each absolute pose Tloc(n)_to_world (where n ranges from 1 to NN), calculate the pose difference between Tloc(n)_to_world and all other Tloc(i)_to_world (where i ≠ n).

[0143] Step 1.2, for each pair of absolute poses Tloc(n)_to_world and Tloc(i)_to_world, calculate the translation error and rotation error, where:

[0144] Translation error: Calculate the Euclidean distance between the translation components (i.e., position components) of two poses. If this distance is less than the translation threshold (e.g., 2 meters), the translation error is considered to be within an acceptable range.

[0145] Rotation error: Calculate the angular difference between the rotational components (i.e., orientation components) of two poses. If this angular difference is less than the rotation threshold (e.g., 5 degrees), the rotation error is considered to be within an acceptable range.

[0146] 1.3 If the translation and rotation errors of Tloc(n)_to_world and Tloc(i)_to_world are both within acceptable ranges, then cast a vote for Tloc(n)_to_world.

[0147] 2. Select the absolute pose that received the most votes and the poses that are related to that absolute pose in the voting:

[0148] 2.1 Count the number of votes for each Tloc(n)_to_world, that is, the number of times it meets the translation and rotation error requirements compared with other pose transformations;

[0149] 2.2 Select the Tloc_to_world with the most votes as the output pose of this module, i.e., the optimal absolute pose mentioned above;

[0150] 2.3, all Tloc_to_world points that vote for the best absolute pose of this final output are taken as interior points (i.e. pose transformations that meet the conditions);

[0151] Through the above process, abnormal poses can be filtered out, and the pose transformation that best conforms to the overall consistency can be selected as the optimal absolute pose. The related poses that have a voting relationship with the optimal absolute pose can be used as the initial value for pose graph optimization, thereby enabling the optimization process to converge faster.

[0152] S1032, Based on the optimal absolute pose and poses that are associated with it through voting, construct an EM pose graph, and use the EM pose graph to optimize the absolute pose. Specifically, this includes the following steps:

[0153] Step 1: Construct the EM pose graph using each absolute pose as a node and the relative relationships between absolute poses as edges.

[0154] Step 2, initialize parameters. Use the pose of the filtered output in step S1031 above as the initial pose of Opted; and initialize the denoising parameters. Optionally, initialize the denoising weight parameter w_k = 1.0.

[0155] Step 3, Expected Steps: Based on the estimation results of the current pose graph, evaluate the matching quality of each edge and determine its contribution to the overall optimization; based on the calculated posterior probability, model the noise of each pose and edge, and update the denoising parameters.

[0156] Step 4: Define the optimization objective and perform optimization. The objective is to minimize the error of all edges in the entire graph, that is, to minimize the sum of the residuals of all relative pose constraints. The residuals can be expressed as the difference between the actual observed relative pose and the estimated pose of the nodes in the graph.

[0157] Step 5, Iterative Optimization: Repeat the iterations between the expectation step and the maximization step to gradually optimize the pose graph. In each iteration, recalculate the posterior probability and the objective function to gradually approach the global optimum. When the convergence condition is met, stop the iteration; the pose in this state is the final optimized pose.

[0158] Specifically, in this embodiment, the optimization process can be implemented using the following formula:

[0159]

[0160]

[0161] Where Loc_m is the pose output by visual localization in the m-th frame, Opted is the localization pose optimized based on the constraints of visual localization in the M-th frame, and opted -1 *Loc m *Identity represents the constraint between the pose output by visual localization and the optimized localization pose. k This represents the weight of the edge that provides constraints for each pose.

[0162] After obtaining the absolute pose through the above steps S1031 and S1032, considering the possibility that the pose may be inaccurate due to erroneous frames, the absolute pose is further optimized by constructing an EM pose graph. Compared with traditional optimization methods, noise in the pose can be removed, further improving the accuracy of the localization results.

[0163] The visual positioning process provided by steps S101 to S103 above, based on an unstructured map in a geographic coordinate system that does not include 3D point alignment, eliminates the need for PNP calculations. Visual positioning pose can be obtained through 2D image matching and relative pose estimation based on deep learning algorithms. Since the visual positioning map lacks 3D structure, server memory usage is significantly reduced, and map updates and expansions are simpler and more efficient. Furthermore, because only the image needs to be encrypted, without encrypting 3D points, map data privacy protection is easier to achieve. In addition, the visual positioning method provided in this application significantly improves positioning efficiency and success rate.

[0164] This embodiment also provides a low-cost visual positioning system based on an unstructured map, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0165] Figure 9 This is a structural block diagram of a low-cost visual positioning system based on an unstructured map according to an embodiment of this application. The system includes: a map building module 90, a search module 91, and a positioning module 92, wherein:

[0166] The map building module 90 is used to build an unstructured map with the poses of each image aligned to the geographic coordinate system based on the initial mapping data with gravity alignment. The initial mapping data includes multiple trajectories, each of which includes an image and image pose and GPS positioning information.

[0167] The search module 91 is used to receive the image to be located and determine candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map.

[0168] The localization module 92 is used to obtain the relative pose between the image to be localized and each candidate map frame through a deep learning algorithm, and to obtain the visual localization result based on the relative pose and the pose of the candidate map frame aligned to the geographic coordinate system in the unstructured map.

[0169] The system described above, based on an unstructured map in a geographic coordinate system that does not include 3D point alignment, eliminates the need for PNP computation. Visual positioning pose can be obtained through 2D image matching and relative pose estimation using deep learning algorithms. Since the visual positioning map lacks 3D structure, server memory usage is significantly reduced, and map updates and expansions are simpler and more efficient. Furthermore, because only the image needs encryption, not the 3D points, map data privacy is easier to protect. In addition, the visual positioning method provided in this application significantly improves positioning efficiency and success rate.

[0170] also, Figure 10 This is a schematic diagram of another low-cost visual positioning system based on an unstructured map according to an embodiment of this application.

[0171] In one embodiment, Figure 11 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 11 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 11 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a low-cost visual positioning method based on unstructured maps, and the database stores data.

[0172] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A low-cost visual localization method based on unstructured maps, characterized in that, The method includes: Based on the initial mapping data aligned by gravity, an unstructured map is constructed with the poses of each image aligned to the geographic coordinate system. Receive the image to be located, and determine candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map. The relative pose between the image to be located and each candidate map frame is obtained by using a deep learning algorithm. Based on the relative pose and the pose of the candidate map frame aligned to the geographic coordinate system in the unstructured map, the absolute pose corresponding to the image to be located is obtained. The process of constructing an unstructured map with the pose of each image aligned to the geographic coordinate system based on the initial mapping data with gravity alignment includes: acquiring the initial mapping data collected by a smart handheld mobile terminal, wherein the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes the gravity alignment pose and GPS positioning information corresponding to the image; Using GPS positioning information as constraints, the initial mapping data is optimized using 3DOF, and the horizontal direction and heading angle of each trajectory are aligned to obtain the initial alignment data relative to the geographic coordinate system in X, Y, pitch, roll and yaw. An elevation map is constructed based on the initial aligned data. The elevation map is then optimized using the maximum spanning tree method, aligning the elevation values ​​of each trajectory in the initial aligned data to the same coordinate system. This results in aligned data where all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data are relative values. An unstructured map is then constructed based on the aligned data.

2. The method according to claim 1, characterized in that, Based on the similarity between the image to be located and each image frame in the unstructured map, candidate map frames are determined in the unstructured map, including: The image to be located is matched one by one with all the map frames in the unstructured map using a deep learning matching algorithm, and the candidate map frames are determined based on the similarity obtained from the matching. Alternatively, obtain the GPS information corresponding to the image to be located, determine the target area in the unstructured map based on the GPS information, and select the first candidate frame in the target area. The similarity between the image to be located and each of the first candidate frames is obtained by using a deep learning matching algorithm, and the candidate map frame is determined in the first candidate frames based on the similarity.

3. The method according to claim 1, characterized in that, After obtaining the absolute pose corresponding to the image to be localized, the method further includes: For each absolute pose, calculate the pose difference between it and other absolute poses; Determine whether the pose differences corresponding to each absolute pose are less than a preset error threshold. If so, add a voting mark to the absolute pose based on the number of pose differences that are less than the preset error threshold. The absolute pose with the most votes is recorded as the optimal absolute pose, and the associated poses that have a voting relationship with the optimal absolute pose are obtained. An EM pose graph is constructed based on the optimal absolute pose and the associated pose. The optimized absolute pose is obtained by optimizing the EM pose graph. Based on the optimized absolute pose, the visual localization result corresponding to the image to be localized is returned.

4. The method according to claim 3, characterized in that, The method further includes: If multiple absolute poses have the same voting label, determine whether the rotation errors of these multiple absolute poses are inconsistent. If so, select the absolute pose with the smallest rotation error as the optimal absolute pose. If not, determine whether the translation errors of the multiple absolute poses are inconsistent. If so, select the absolute pose with the smallest translation error as the optimal absolute pose.

5. The method according to claim 1, characterized in that, The unstructured map comprises a series of images, each aligned to a six-degree-of-freedom pose in geographic coordinates.

6. The method according to claim 5, characterized in that, The elevation map is optimized using a maximum spanning tree, aligning the elevation values ​​of each trajectory in the initial aligned data to obtain aligned data including: The maximum spanning tree algorithm is applied to the elevation map to obtain the maximum spanning tree model corresponding to the elevation values ​​between nodes; The root node of the maximum spanning tree model is determined, and the elevation value is propagated based on the maximum spanning tree model to calculate the elevation difference between other nodes and the root node, wherein the elevation value of the root node is set to 0. Based on the elevation difference between each node and the root node, the elevation values ​​of all nodes are aligned to obtain aligned data in the geographic coordinate system, where all six degrees of freedom are aligned.

7. The method according to claim 6, characterized in that, Aligning the elevation values ​​of each trajectory in the initial alignment data to the same coordinate system, and obtaining the alignment data, aligning multiple trajectories to the geographic coordinate system in six degrees of freedom (X, Y, Z, Pitch, Roll, and Yaw), the method further includes: Based on the poses of successfully looped images under different trajectories during visual loop closure detection, the pose of a virtual reference frame is determined. The orientation of the virtual reference frame pose is the orientation of the image under its original trajectory after 3DOF optimization, and the position of the virtual reference frame pose is the average of the pose translation of the image under the original trajectory and all successfully looped trajectories. With the virtual reference frame pose fixed, the alignment data is optimized using 6DOF to obtain the optimized alignment data. The unstructured map is constructed based on the optimized aligned data.

8. A low-cost visual positioning system based on unstructured maps, characterized in that, The system includes: a map building module, a search module, and a positioning module, wherein: The map building module is used to construct an unstructured map with the poses of each image aligned to the geographic coordinate system based on the initial mapping data with gravity alignment. The search module is used to receive an image to be located and determine candidate map frames in the unstructured map based on the similarity between the image to be located and each image frame in the unstructured map. The positioning module is used to obtain the relative pose between the image to be located and each candidate map frame through a deep learning algorithm, and to obtain a visual positioning result based on the relative pose and the pose of the candidate map frame aligned to the geographic coordinate system in the unstructured map. The process of constructing an unstructured map with the pose of each image aligned to the geographic coordinate system based on the initial mapping data with gravity alignment includes: acquiring the initial mapping data collected by a smart handheld mobile terminal, wherein the initial mapping data includes multiple trajectories, each trajectory includes multiple tuple data sequences, and each tuple data sequence includes the gravity alignment pose and GPS positioning information corresponding to the image; Using GPS positioning information as constraints, the initial mapping data is optimized using 3DOF, and the horizontal direction and heading angle of each trajectory are aligned to obtain the initial alignment data relative to the geographic coordinate system in X, Y, pitch, roll and yaw. An elevation map is constructed based on the initial aligned data. The elevation map is then optimized using the maximum spanning tree method, aligning the elevation values ​​of each trajectory in the initial aligned data to the same coordinate system. This results in aligned data where all six degrees of freedom are aligned on the geographic coordinate system. The Z-axis data in the aligned data are relative values. An unstructured map is then constructed based on the aligned data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images