Geometrically assisted visual positioning method and system

Abstract

A geometric structure aided visual localization method and a system implementing the method are provided. The method includes retrieving a 3D map of initialization location data, the 3D map being constrained by visual structures and geometric structures and modeled by a Gaussian mixture model of a set of Gaussian distributions with mapped landmarks; acquiring a series of real-time image frames by a camera of a mobile system; and for each real-time image frame: extracting local features from the real-time image frame; predicting a camera pose corresponding to the real-time image frame; creating a key frame by tracking the predicted camera pose in the 3D map; identifying temporally visible landmarks with respect to the created key frame; acquiring 2D-3D correspondences between the local features and the temporally visible landmarks; and localizing the mobile system by estimating a state of the real-time image frame based on the 2D-3D correspondences.

Description

Technical Field

[0001] This invention relates generally to visual positioning, and more specifically, to a geometrically assisted visual positioning method and system. Background Technology

[0002] Positioning systems play a crucial role in autonomous navigation and virtual / augmented reality. Despite the existence of various existing technologies based on different sensor configurations, Global Navigation Satellite Systems (GNSS) and Inertial Navigation Systems (INS) are still widely used for positioning in autonomous systems. However, their performance is inconsistent in urban areas and completely fails in indoor environments. An alternative is to utilize three-dimensional (3D) LiDAR devices. 3D LiDAR devices scan the scene structure and match it with a pre-built point cloud map. However, due to the limited appearance information provided by a single scan, it is not easy to identify locations from existing databases to recover the global position using LiDAR scans. Furthermore, GNSS / INS or LiDAR systems with the ability to recover centimeter-level sensor attitude are generally expensive, limiting their widespread application in navigation or other applications.

[0003] Traditional vision-based methods match query images with sparse 3D scene models rich in visual features. Camera pose is then estimated using the 3D-to-2D correspondence. Compared to GNSS / INS or LiDAR-based systems, vision-based positioning systems may be more cost-effective. However, this comes at the cost of being highly sensitive to various conditions due to visual perception and the inability to directly acquire measurement information from visual sensors. These two drawbacks make general vision-based positioning systems less robust and limit their application scenarios.

[0004] To alleviate the problems in vision-based localization systems, some methods introduce pre-generated dense map information, which helps constrain visual structures, leading to better pose estimation results. However, compared to traditional vision-based pipelines, these systems typically require more computational resources, necessitating high-spec desktop computers, thus limiting their application on large-scale, especially embedded, devices. Therefore, developing a low-cost localization system that is robust, accurate, and scalable remains a significant challenge. There is also a desire for a potentially computationally efficient vision-based localization method and system that can be applied to cutting-edge computing devices. Summary of the Invention

[0005] According to one aspect of the present invention, a geometry-assisted visual positioning method for a navigation mobile system is provided. The method includes: retrieving a 3D map with respect to initialized location data. The 3D map is constrained by visual and geometric structures and consists of landmarks with mappings. The system is modeled using a Gaussian mixture model with a Gaussian distribution; a series of real-time image frames are acquired using a camera of the mobile system; and for each real-time image frame: one or more local features are extracted from the real-time image frame; the camera pose corresponding to the real-time image frame is predicted; a keyframe is created by tracking the predicted camera pose in a 3D map; one or more temporally visible landmarks are identified relative to the created keyframes; one or more two-dimensional to three-dimensional (2D-3D) correspondences between the local features and the temporally visible landmarks are obtained; and the mobile system is localized by estimating the state of the real-time image frames based on one or more 2D-3D correspondences.

[0006] According to another aspect of the present invention, a geometry-assisted visual positioning system for a navigation mobile system is provided. The system includes a camera configured to acquire a series of real-time image frames; and a first processor including a mapping module configured to retrieve a 3D map based on initialized location data. The 3D map is constrained by visual and geometric structures and consists of landmarks with mappings. The system is modeled using a set of Gaussian mixture models with Gaussian distributions; and a second processor including a localization module configured to process each real-time image frame by: extracting one or more local features from the real-time image frame; predicting the camera pose corresponding to the real-time image frame; creating a keyframe by tracking the predicted camera pose in a 3D map; identifying one or more temporally visible landmarks relative to the created keyframe; obtaining one or more two-dimensional to three-dimensional (2D-3D) correspondences between the local features and the temporally visible landmarks; and localizing the mobile system by estimating the state of the real-time image frame based on one or more 2D-3D correspondences.

[0007] Preferably, the camera attitude is predicted through a visual inertial measurement process, which includes: acquiring real-time inertial measurement data through the inertial measurement unit of the mobile system; pre-integrating the real-time inertial measurement data into a single relative motion constraint; and estimating the camera attitude based on the single relative motion constraint.

[0008] Preferably, the one or more temporally visible landmarks are identified through the following steps: dividing the 3D map into a set of voxels to form a voxelized map, wherein each mapped landmark intersects with one or more voxels; projecting multiple rays from the optical center of the created keyframe into the voxelized map; for each projected ray, locating one or more voxels along the projected ray; and identifying the one or more mapped landmarks that intersect with the located voxels as temporally visible landmarks relative to the created keyframe.

[0009] The provided visual positioning method and system can receive input data from low-cost sensor suites (such as visual-inertial sensors) and utilize other modalities, such as high-definition maps generated by expensive LiDAR devices and integrated navigation systems. Therefore, it can leverage the advantages of both geometric and visual structures without requiring an initial attitude guidance system. This allows for a good balance between positioning accuracy and computational efficiency. Attached Figure Description

[0010] Various aspects of this disclosure may be readily understood from the following detailed description and with reference to the accompanying drawings. These illustrations are not necessarily drawn to scale. In the drawings and detailed description, common reference numerals may be used to indicate the same or similar components;

[0011] Figure 1 A block diagram of a geometry-assisted visual positioning system according to an embodiment of the present invention is shown;

[0012] Figure 2 Showing Figure 1 Example sensor setup for a geometry-assisted vision positioning system is shown;

[0013] Figure 3 A flowchart of a positioning method according to an embodiment of the present invention is shown;

[0014] Figure 4 An example visual-inertial bundled adjustment pipeline according to an embodiment of the present invention is shown; and

[0015] Figure 5 Showing Figure 4 The factor diagram shown represents the visual inertial binding adjustment pipeline. Detailed Implementation

[0016] In the following description, preferred examples of this disclosure are presented as illustrative rather than limiting embodiments. Certain specific details may be omitted in order to avoid making this disclosure redundant; however, this disclosure is prepared to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0017] Figure 1 A block diagram of a geometry-assisted visual positioning system according to an embodiment of the present invention is shown. Figure 2 Showing Figure 1 Example of sensor setup for a visual positioning system.

[0018] As shown in the figure, the visual positioning system can be completely decomposed into a mapping module 11 and a positioning module 12. The mapping module 11 can be run offline in advance to generate an accurate, pre-localizable map that is expected to be fully reusable in long-term positioning. The pre-localizable map can be constructed offline using more sophisticated sensors. To this end, the visual positioning system can receive an input trajectory estimated from an integrated navigation system that combines information from a laser scanner, camera system, and GNSS / INS (in function block 102: Integrated Navigation). Furthermore, geometric and visual features are extracted from the laser and visual data, respectively (in function block 104: Geometric / Visual Feature Extraction). Finally, these geometric and visual features serve as map elements for constructing the localizable map. Under the estimated pose, features within the local frame are transformed and aggregated to form the localizable map (in function block 106: Map Aggregation).

[0019] Having a prior localizable map, the localization module 12 can operate independently. The localization module 12 receives query images and / or inertial measurement data from a visual-inertial sensor suite. The visual-inertial sensor suite may include a camera system and an inertial measurement unit (IMU). For each input query image, (in function block 108: Local Feature Extraction) local features are extracted to represent the current frame. (in step 110: Pre-integration) pre-integration is performed on the inertial measurement data obtained from the previous frame to obtain the relative transformation and corresponding covariance to estimate the previous attitude. (in function block 112: Inter-frame Tracking) Local features are first matched, using the previous attitude as an initial guess to track the previous frame. (in function block 114: Local Feature Association) Local features are associated with map elements obtained from the localizable map using relatively accurate attitude estimation to generate some landmarks. Finally, (in function block 116: Co-optimization) Optimizable parameters are co-tuned to refine the attitude and inertial parameters.

[0020] like Figure 2 As shown, the mapping module 11 may include a high-standard sensor suite, including but not limited to a laser scanner 22, a camera system 24, and a GNSS / INS 26, to acquire high-precision and high-density data to build previously localizable and reusable maps. The positioning module 12 may include an inexpensive visual-inertial (VI) sensor 28 (e.g., a VI sensor equipped on a smartphone) to acquire visual and inertial measurement data, thereby minimizing costs.

[0021] The mapping module 11 and the positioning module 12 may be implemented by one or more processors (not shown), which may be CPUs, MCUs, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or any suitable programmable logic devices, configured or programmed to perform mapping and positioning. The positioning system may also include a memory unit (not shown), which may include volatile memory units (such as RAM), non-volatile units (such as ROM, EPROM, EEPROM, and flash memory), or both, or any medium or device suitable for storing instructions, code, and / or data.

[0022] Figure 3 A flowchart of a visual localization method according to an embodiment of the present invention is shown. As shown, the visual localization method may include:

[0023] S301: Retrieve a 3D map of the initial location data The 3D map is constrained by visual and geometric structures and consists of landmarks with mappings. Modeling a Gaussian mixture model with a set of Gaussian distributions;

[0024] S302: A series of real-time image frames are acquired by the camera of the mobile system;

[0025] S303: For each real-time image frame, extract one or more local features;

[0026] S304: Predict the camera pose corresponding to the real-time image frame;

[0027] S305: Create keyframes by tracking the predicted camera pose in a 3D map;

[0028] S306: Identify one or more temporally visible landmarks relative to the created keyframe;

[0029] S307: Obtain one or more 2D-to-3D correspondences between local features and temporally visible landmarks; and

[0030] S308: Localize a mobile system by estimating the state of a real-time image frame based on one or more 2D-3D correspondences.

[0031] In step S301, a visual structure can be constructed using Simultaneous Localization and Mapping (SLAM) or Structure of Motion (SfM) methods, where the selection of landmarks is filtered by the number of frames in which they can be observed. The geometric structure can be constructed using a point cloud representing the shapes of landmarks in a 3D map.

[0032] The Gaussian mixture model is expressed and parameterized as follows:

[0033]

[0034] in, It is the 3D location of the i-th mapped landmark. Given the j-th Gaussian distribution Landmark Location The likelihood; The weights of the j-th Gaussian distribution for each landmark are... and These are the mean and covariance of the j-th Gaussian distribution, respectively.

[0035] The Gaussian distribution of each mapped landmark is parameterized as follows: ,in ,in, , It is a singular value and has Relationship, , It is an eigenvector.

[0036] In step 304, the camera attitude can be predicted through a visual inertial measurement process, including: acquiring real-time inertial measurement data through the inertial measurement unit of the mobile system; pre-integrating the real-time inertial measurement data into a single relative motion constraint; and estimating the camera attitude based on the single relative motion constraint.

[0037] Alternatively, camera pose can be predicted through a location recognition process, including: retrieving multiple keyframe candidates with global descriptors based on initial location data; grouping the keyframe candidates into clusters based on their co-visibility with one or more landmarks; matching local features with each cluster to obtain the correspondence between real-time image frames and keyframe candidates; recovering the camera pose using a perspective-n-point (PnP) scheme and a random sampling consistency (RANSAC) method; and fine-tuning the camera pose through interior point matching.

[0038] In step 306, temporally visible landmarks relative to the created keyframe can be identified by the following steps: projecting the mapped landmarks onto the created keyframe; and filtering the projected landmarks based on one or more visibility criteria to identify those landmarks that pass one or more projections of the visibility criteria as one or more temporally visible landmarks relative to the created keyframe.

[0039] Alternatively, temporally visible landmarks can be identified relative to the created keyframe by the following steps: dividing the 3D map into a set of voxels to form a voxelized map, where each mapped landmark intersects with one or more voxels; projecting multiple rays from the optical center of the created keyframe into the voxelized map; for each projected ray, locating one or more voxels along the projected ray; and identifying the mapped landmarks that intersect with the located voxels as temporally visible landmarks relative to the created keyframe.

[0040] Each projected ray can be represented as ,in It is the location of the optical center in world coordinates. It is the normalized direction vector of the projected ray. It is possible Calculation, where These are the pixel positions of each local feature. Represents the camera projection function. It is a rotation matrix from world coordinates to camera coordinates. The position of each temporally visible landmark is parameterized as follows: ,in Indicates the voxel position of the location. It is the inverse depth of the time-series visible landmarks.

[0041] After projecting a ray onto a voxelized map, each voxel in the map is examined to determine if it contains geometric components (or a Gaussian distribution of landmarks) that can be associated with local features. Assuming the geometry is parameterized as a 3D Gaussian distribution (μ, Σ), it can be decomposed using singular value decomposition (SVD): , , , ray parameters It can be whitened as:

[0042] ,

[0043] ,

[0044] And ray parameters can be used. The Mahalanobious distance between them determines whether the geometric component intersects with the ray.

[0045] After applying the whitening transformation, the ray parameters become:

[0046]

[0047] The distance to Maharanobis is given by the following formula:

[0048]

[0049] If the Mahalanobis distance is within the threshold range, the inverse depth can be obtained as follows:

[0050]

[0051] Optionally, the location of the identified temporally visible landmarks can be further optimized through a nearest-point interaction (ICP) scheme, which includes: selecting a set of nearest projected Gaussian distributions as association candidates for each temporally visible landmark; optimizing the location of the temporally visible landmarks by iterating among the association candidates; and selecting the final association candidate that results in the minimum reprojection error.

[0052] In step S308, a given total energy function can be minimized by solving a least-squares optimization problem. To estimate the state of real-time image frames, the total energy function is defined as follows:

[0053]

[0054] Among them, E visual E strucure and E prior These are the visual term, structural term, and prior term of the total energy function, respectively.

[0055] Visual item E visual Given by the following formula:

[0056]

[0057] Where ρ(.) is the probability density function; It is the reprojection residual term projected from the i-th mapped landmark onto the k-th image frame, which depends on the predicted camera pose of the k-th image frame. 3D location of the i-th mapped landmark ;and It is the covariance of the 2D Gaussian distribution of the projection of the i-th mapped landmark onto the k-th image frame. It is the set of associations for visual items.

[0058] Generally speaking, the reprojection residual term Given by the following formula:

[0059]

[0060] in, It is the 2-D position of the visual feature corresponding to the projection of the i-th temporally visible landmark onto the k-th image frame. and These are rotation and translation matrices, used to project landmarks from the world coordinates of the 3D map to the k-th image frame.

[0061] When predicting camera pose using a visual inertial measurement process, the reprojection residual term It can be represented as:

[0062]

[0063] in, and These are the rotation and translation matrices used to transform the world coordinates of the 3D map to the body coordinates relative to the inertial measurement unit of the k-th image frame; and These are rotation and translation matrices, used to transform the body coordinates of the inertial measurement unit to the k-th image frame.

[0064] Structure item It can be represented as:

[0065]

[0066] in, It is the associative set of structural items. It is the j-th Gaussian distribution, used to represent the landmark of the i-th mapping. It is a structural residual term. It is a residual term in a degraded state. is a function used to indicate whether the landmark of the i-th mapping is degenerate, and it is defined as:

[0067]

[0068] For a given likelihood of the i-th landmark, the structural residual term Given by the following formula:

[0069]

[0070] in, It is the three-dimensional position of the i-th mapped landmark. and These are the mean and covariance of the j-th Gaussian distribution, respectively.

[0071] Residual terms of degradation Given by the following formula:

[0072]

[0073] in, It is the 3D location of the i-th mapped landmark. It is the mean of the j-th Gaussian distribution; It is the first eigenvector of the j-th Gaussian distribution. ,in It is the predefined variance of the geometric structure.

[0074] Priors It can be defined as the initial residual term, given by the following formula:

[0075]

[0076] in, It is a preset camera pose based on the initial position data. That is the actual posture.

[0077] Therefore, the optimized landmark location can be represented as:

[0078] ,

[0079] in, It is the projected residual term. It is the pose parameterization of the k-th image frame. It is the predefined variance of the feature corresponding to the i-th mapped landmark.

[0080] The least squares optimization problem can be solved using the Levenberg-Marquardt method, which provides solutions for the following systems:

[0081] ,

[0082]

[0083] Where J and r are the stacked Jacobian matrix and residual, respectively, and W is the weight matrix formed by stacking the inverse covariances of the residual terms. It is the damping factor of the Levenberg-Marquardt method.

[0084] In situations where certain states cannot be accurately estimated, especially when inertial measurements are incorporated, a batch-processing yet efficient optimization problem can be formulated, which can be solved in an average of 10 milliseconds. First, the batch-processing optimization problem can be viewed as a bundle adjustment (BA), which can be decoupled into visual-inertial (or motion-only, unstructured) BA and structure-only BA.

[0085] Figure 4 An example visual-inertial (BA) pipeline 400 according to an embodiment of the present invention is shown. Figure 5A factor graph representation of the visual-inertial (BA) pipeline is shown. The BA pipeline 400 may include a frame tracking module 401, an edge detection module 402, a co-optimization module 403, and a parameter update module 404. Although the decoupling scheme accelerates optimization, iteratively relinearizing visual factors may be computationally inefficient, while real-time localization already provides relatively accurate estimates. Figure 4 and Figure 5 As shown, visual factors obtained through instantaneous localization can be marginalized as prior poses instead of using the original visual factors. In other words, after optimizing the state of a real-time image frame, visual factors can be "marginalized" to form prior terms of the real-time image frame pose. This eliminates all visual factors in the optimization, significantly reducing the computational resources required for visual-inertial binding adjustments.

[0086] More specifically, the estimated values ​​and their corresponding covariance retained in the instantaneous localization are used as priors in the bundled adjustment. The camera pose and inertial parameters are jointly optimized and adjusted, with the energy function including the following priors:

[0087] ,

[0088]

[0089] in, and These are the rotation and translation terms adjusted by visual inertial binding.

[0090] Therefore, the camera pose can be optimized by minimizing the objective function:

[0091]

[0092] Where F is the keyframe in the current window, and its modulus |F| can be set to 15.

[0093] Assuming the attitude estimation reaches a local optimum, the state update... Therefore, the prior error state can be expressed as:

[0094]

[0095] Then, the overall objective function can be changed to:

[0096]

[0097] The Levenberg-Marquardt method can then be used to solve this problem. The advantage of this approach is that it does not require relinearizing the visual factors, which may be computationally more efficient.

[0098] The selected and described embodiments are intended to best explain the principles of the invention and its practical application, enabling those skilled in the art to understand the various embodiments of the invention and the various modifications suitable for specific uses. Although the methods disclosed herein are described as specific operations performed in a particular order, it will be understood that these operations may be combined, divided, or rearranged to form equivalent methods without departing from the teachings of the invention. Therefore, the order and grouping of operations are not limiting unless expressly indicated herein. While the apparatuses disclosed herein are described in accordance with specific structures, shapes, materials, material compositions, and relationships, these descriptions and illustrations are not limiting. Modifications may be made to adapt particular circumstances to the objectives, spirit, and scope of the invention. All such modifications are considered to be within the scope of the claims appended herein.

Claims

1. A geometry-assisted visual positioning method for navigation mobile systems, characterized in that, include: Retrieve 3D maps based on initial location data The 3D map is constrained by visual and geometric structures and consists of landmarks with mappings. Modeling a Gaussian mixture model with a set of Gaussian distributions; A series of real-time image frames are acquired through the camera of the mobile system; as well as For each of the real-time image frames: Extract one or more local features from the real-time image frame; Predict the camera pose corresponding to the real-time image frame; Keyframes are created by tracking the predicted camera pose in the 3D map; Identify one or more temporally visible landmarks relative to the created keyframe; Obtain one or more two-dimensional to three-dimensional (2D-3D) correspondences between the local features and the temporally visible landmarks; as well as The mobile system is located by estimating the state of the real-time image frame based on one or more 2D-3D correspondences.

2. The geometric structure-assisted visual positioning method according to claim 1, characterized in that, The camera pose is predicted through a position recognition process, which includes: Retrieve multiple keyframe candidates with global descriptors based on initialized location data; The keyframe candidates are grouped into clusters based on their co-visibility with one or more landmarks; The local features are matched with each of the clusters to obtain the correspondence between the real-time image frames and the keyframe candidates; The camera pose was recovered using the perspective-n-point (PnP) scheme and the random sampling consistency (RANSAC) method; and The camera pose is finely adjusted by interior point matching.

3. The geometric structure-assisted visual positioning method according to claim 1, characterized in that, The camera pose can be predicted through a visual inertial measurement process, which includes: Real-time inertial measurement data is acquired through the inertial measurement unit of the mobile system; The real-time inertial measurement data is pre-integrated into a single relative motion constraint; and The camera pose is estimated based on the single relative motion constraint.

4. The geometric structure-assisted visual positioning method according to claim 2 or 3, characterized in that, The step of identifying one or more of the temporally visible landmarks relative to the created keyframe includes: Project the mapped landmarks onto the created keyframes; and Landmarks of the projection are filtered based on one or more visibility criteria to identify landmarks of one or more projections that pass through the visibility criteria as one or more of the time-series visible landmarks relative to the created keyframe.

5. The geometric structure-assisted visual positioning method according to claim 2 or 3, characterized in that, The step of identifying one or more of the temporally visible landmarks relative to the created keyframe includes: The 3D map is divided into a set of voxels to form a voxelized map, wherein each mapped landmark intersects with one or more voxels; Multiple rays are projected from the optical center of the created keyframe onto the voxelized map; For each projected ray: Positioning one or more voxels along the projected ray; and Landmarks that intersect with the voxels of the location are identified as temporally visible landmarks relative to the created keyframes.

6. The geometric structure-assisted visual positioning method according to claim 4, characterized in that, The step of estimating the state of the real-time image frame includes solving a least-squares optimization problem to minimize a given total energy function. The total energy function The definition is as follows: Among them, E visual E strucure and E prior These are the visual term, structural term, and prior term of the total energy function, respectively.

7. The geometric structure-assisted visual positioning method according to claim 6, characterized in that, The method also includes further optimizing the location of identified temporally visible landmarks through a nearest-point interaction (ICP) scheme.

8. The geometric structure-assisted visual positioning method according to claim 7, characterized in that, The nearest point interaction (ICP) scheme includes: For each time-series visible landmark, select a set of the nearest projected Gaussian distributions as association candidates; The location of the temporally visible landmarks is optimized by iterating among the associated candidates; and Select the final association candidate that results in the minimum reprojection error.

9. The geometric structure-assisted visual positioning method according to claim 5, characterized in that, The step of estimating the state of the real-time image frame includes solving a least-squares optimization problem to minimize a given total energy function. The total energy function The definition is as follows: Among them, E visual E strucure and E prior These are the visual term, structural term, and prior term of the total energy function, respectively.

10. The geometric structure-assisted visual positioning method according to claim 9, characterized in that, The method also includes further optimizing the location of identified temporally visible landmarks through a nearest-point interaction (ICP) scheme.

11. The geometric structure-assisted visual positioning method according to claim 10, characterized in that, The nearest point interaction (ICP) scheme includes: For each time-series visible landmark, select a set of the nearest projected Gaussian distributions as association candidates; The location of the temporally visible landmarks is optimized by iterating among the associated candidates; and Select the final association candidate that results in the minimum reprojection error.

12. The geometric structure-assisted visual positioning method according to claim 1, characterized in that, The method also includes constructing the 3D map from a point cloud generated by a lidar (LiDAR) system.

13. The geometric structure-assisted visual positioning method according to claim 1, characterized in that, The initial location data is obtained through a GNSS / INS.

14. A geometry-assisted visual positioning system for a navigation mobile system, characterized in that, include: A camera configured to acquire a series of real-time image frames; The first processor includes a mapping module configured to retrieve a 3D map based on initialized location data. The 3D map is constrained by visual and geometric structures and consists of landmarks with mappings. Modeling a Gaussian mixture model with a set of Gaussian distributions; and And a second processor including a positioning module configured to process each real-time image frame through the following steps: Extract one or more local features from the real-time image frame; Predict the camera pose corresponding to the real-time image frame; Keyframes are created by tracking the predicted camera pose in the 3D map; Identify one or more temporally visible landmarks relative to the created keyframe; Obtain one or more two-dimensional to three-dimensional (2D-3D) correspondences between the local features and the temporally visible landmarks; as well as The mobile system is located by estimating the state of the real-time image frame based on one or more 2D-3D correspondences.

15. The geometric structure-assisted visual positioning system according to claim 14, characterized in that, The positioning module is also configured to predict the camera pose corresponding to the real-time image frame through the following steps: Retrieve multiple keyframe candidates with global descriptors based on initialized location data; The keyframe candidates are grouped into clusters based on their co-visibility with one or more landmarks; The local features are matched with each of the clusters to obtain the correspondence between the real-time image frames and the keyframe candidates; The camera pose was recovered using the perspective-n-point (PnP) scheme and the random sampling consistency (RANSAC) method. as well as The camera pose is finely adjusted by interior point matching.

16. The geometric structure-assisted visual positioning system according to claim 14, characterized in that, It also includes an inertial measurement unit configured to acquire real-time inertial measurement data; and wherein the positioning module is further configured to predict the camera pose corresponding to the real-time image frame through the following steps: The real-time inertial measurement data is pre-integrated into a single relative motion constraint; and Camera pose is estimated based on a single relative motion constraint.

17. The geometric structure-assisted visual positioning system according to claim 15 or 16, characterized in that, The localization module is also configured to identify one or more of the temporally visible landmarks relative to the created keyframe through the following steps: Project the mapped landmarks onto the created keyframes; as well as Landmarks of the projection are filtered based on one or more visibility criteria to identify landmarks of one or more projections that pass through the visibility criteria as one or more of the time-series visible landmarks relative to the created keyframe.

18. The geometric structure-assisted visual positioning system according to claim 15 or 16, characterized in that, The localization module is also configured to identify one or more of the temporally visible landmarks relative to the created keyframe through the following steps: The 3D map is divided into a set of voxels to form a voxelized map, wherein each mapped landmark intersects with one or more voxels; Multiple rays are projected from the optical center of the created keyframe onto the voxelized map; For each projected ray: Locate one or more voxels along the projected ray; as well as Landmarks that intersect with the voxels of the location are identified as temporally visible landmarks relative to the created keyframes.

19. The geometric structure-assisted visual positioning system according to claim 17, characterized in that, The positioning module is also configured to minimize a given total energy function by solving a least-squares optimization problem. Estimate the state of the real-time image frame, the total energy function The definition is as follows: Among them, E visual E strucure and E prior These are the visual term, structural term, and prior term of the total energy function, respectively.

20. The geometric structure-assisted visual positioning system according to claim 19, characterized in that, The positioning module is also configured to further optimize the location of identified temporally visible landmarks through a nearest-point interaction (ICP) scheme.

21. The geometric structure-assisted visual positioning system according to claim 20, characterized in that, The positioning module is also configured to perform the nearest point interaction (ICP) scheme through the following steps: For each time-series visible landmark, select a set of the nearest projected Gaussian distributions as association candidates; The location of the temporally visible landmark is optimized by iterating among the associated candidates; as well as Select the final association candidate that results in the minimum reprojection error.

22. The geometric structure-assisted visual positioning system according to claim 18, characterized in that, The positioning module is also configured to minimize a given total energy function by solving a least-squares optimization problem. Estimate the state of the real-time image frame, the total energy function The definition is as follows: Among them, E visual E strucure and E prior These are the visual term, structural term, and prior term of the total energy function, respectively.

23. The geometric structure-assisted visual positioning system according to claim 22, characterized in that, The positioning module is also configured to further optimize the location of identified temporally visible landmarks through a nearest-point interaction (ICP) scheme.

24. The geometric structure-assisted visual positioning system according to claim 23, characterized in that, The positioning module is also configured to perform the nearest point interaction (ICP) scheme through the following steps: For each time-series visible landmark, select a set of the nearest projected Gaussian distributions as association candidates; The location of the temporally visible landmark is optimized by iterating among the associated candidates; as well as Select the final association candidate that results in the minimum reprojection error.

25. The geometric structure-assisted visual positioning system according to claim 14, characterized in that, It also includes a lidar (LiDAR) system configured to generate point clouds for constructing the 3D map.

26. The geometric structure-assisted visual positioning system according to claim 14, characterized in that, It also includes a GNSS / INS to obtain the initial location data.

Images