Indoor three-dimensional (3D) localization method and system

Abstract

The invention discloses an indoor three-dimensional positioning method and system, and relates to the field of indoor positioning. The method comprises the steps that prior information of a positionin

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of indoor three-dimensional (3D) localization, and in particular to an indoor 3D localization method and system. BACKGROUND

[0002] Indoor visual localization mainly includes visual simultaneous localization and mapping (SLAM) and localization based on a priori map. The visual SLAM has been universally applied to localization and navigation of indoor robots, particularly sweeping robots and service robots. However, due to a failure in continuous tracking of visual features, and particularly a failure in calculation for abrupt change of a pose of the camera-holding pedestrian, this technique cannot be directly applied to localization and navigation of intelligent terminals. Meanwhile, the visual SLAM has an inevitable accumulative error, such that the visual SLAM at a mobile phone side cannot provide a stable and continuous localization result for the pedestrian. Unlike the visual SLAM, visual localization based on priori map information can acquire global 3D location information of the pedestrian, and thus is applied to 3D localization of the pedestrian in a complicated indoor scene.

[0003] In the visual localization based on the priori map information, there are two types of map information, including image label information and accurate 3D model (point cloud) information. For the sake of an accurate 3D location of the pedestrian, visual localization based on a 3D model has been widely used. The visual localization based on the priori 3D model is divided into two stages, namely offline 3D model construction, and online visual feature matching and pose solving. With priori information dependent on a 3D map model, the visual localization based on the priori map information is more accurate than the image label-based method, and can obtain a. six-degrees-of-freedom (6DoF) pose of the carrier, thus realizing accurate 3D location estimation of the pedestrian. However, only by the visual localization, the presence of a visual deviation or a matching failure will affect continuous localization. This cannot realize real-time and uninterrupted localization of a user in a large-scale scene. SUMMARY

[0004] An objective of the present disclosure is to provide an indoor 3D localization method and system, to realize continuous localization, as well as real-time and uninterrupted, localization of a user in a large-scale scene.

[0005] To achieve the above objective, the present disclosure provides the following technical solutions:

[0006] An indoor 3D localization method includes:

[0007] constructing priori information of a localization region in an indoor scene, and constructing a local 3D model of the localization region according to the priori information to form a visual localization priori database, the priori information including a point cloud location and two-dimensional (2D) image features, and the visual localization priori database including 2D image features and 3D point cloud features;

[0008] acquiring, in response to a localization request of a target person, a current scene image with a camera of a mobile phone, and determining a camera pose according to the current scene image and the visual localization priori database, the camera pose being a visual localization result;

[0009] acquiring sensor data of an inertial component of a built-in microelectromechanical system (MEMS) of the mobile phone, and determining a relative localization result according to the sensor data, the sensor data including magnetic field intensity data, angular acceleration data, and acceleration data;

[0010] determining whether the visual localization result is credible based on a visual localization credibility; and

[0011] outputting the visual localization result if yes; or

[0012] outputting the relative localization result if no.

[0013] Optionally, the constructing a local 3D model of the localization region according to the priori information to form a visual localization priori database specifically includes:

[0014] reconstructing, for the large-scale indoor scene, small 3D models piecewise according to the priori information;

[0015] correcting scales of the small 3D models through a total station, and splicing scale-corrected 3D models based on manual surveying and an iterative closest point (ICP) method to form the local 3D model of the localization region; and

[0016] storing the 2D image features and the 3D point cloud features according to the local 3D model to form the visual localization priori database.

[0017] Optionally, the determining a camera pose according to the current, scene image and the visual localization priori database specifically includes: [ 0018] acquiring 2D features of the current scene image;

[0019] comparing and associating the 2D features with the 3D point cloud features, and determining feature matching pairs between the 2D features and the 3D point cloud features with a random sample consensus (RANSAC) algorithm; and

[0020] obtaining interior points of the feature matching pairs with a perspective-n-point (PnP) algorithm to determine the camera pose.

[0021] Optionally, the determining a relative localization result according to the sensor data specifically includes:

[0022] determining a stride length at a current moment according to acceleration data in one step of the target person;

[0023] acquiring a current location point of the target person, and determining a pose of the target person at the cun-ent moment according to the current location point and the sensor data; and

[0024] determining the relative localization result of the target person according to the stride length at the current moment, the pose at the current moment and a location at a previous moment.

[0025] Optionally, when the target person is a rescue worker, a pose of the rescue worker at the current moment is determined according to a recursive equation.

[0026] Optionally, the recursive equation is expressed as:

[0027] 4 + At

[0028] where, k is a l ocation of the target person at a moment k, vZ is a speed of the target Cn n person at we moment k, o k is a pose matrix of the target person at the moment k, w is a V location of the target person at a moment k-1, is a speed of the target person at the moment k-1, is a. pose matrix of the target person at the moment k-1, * is acceleration data of the -,4 target person at the moment k, 4 is angular acceleration, data of the target person at the b moment k, « isa zero offset of an accelerometer for acquiring the acceleration data, 8 is a zero offset of a gyroscope for acquiring the angular acceleration data, is a time interval, is a gravity vector of a navigation coordinate system, and &is an antisymmetric matrix.

[0029] An indoor 3D localization system includes:

[0030] a visual localization priori database formation module configured to construct priori information of a localization region in an indoor scene, and construct a local 3D model of the localization region according to the priori information to form a visual localization priori database, the priori information including a point cloud location and 2D image features, and the visual localization priori database including 2D image features and 3D point cloud features;

[0031] a camera pose determination module configured to acquire, in response to a localization request of a target person, a current scene image with a camera of a mobile phone, and determine a camera pose according to the current scene image and the visual localization priori database, the camera pose being a visual localization result;

[0032] a MEMS-based relative localization module configured to acquire sensor data of an inertial component of a built-in MEMS of the mobile phone, and determine a relative localization result according to the sensor data, the sensor data including magnetic field intensity data, angular acceleration data, and acceleration data;

[0033] a determining module configured to determine whether the visual localization result is credible based on a visual localization credibility;

[0034] a visual localization result output module configured to output the visual localization result; and

[0035] a relative localization result output module configured to output the relative localization result.

[0036] According to specific embodiments of the present disclosure, the present disclosure has the following technical effects: The indoor 3D localization method and system provided by the present disclosure assist the visual localization process with the inertial component of the built-in MEMS of the mobile phone. By constructing the 3D model of the localization region with the acquired priori information, acquiring the feature matching submap, and forming the visual localization priori database, the present disclosure performs matching unnecessarily in a global map, and improves matching efficiency in visual localization. Meanwhile, when the visual localization is failed, the present disclosure takes the MEMS-based relative localization result as an output of the localization system to ensure continuous localization. The present disclosure performs fusion based on the built-in MEMS and the camera of the mobile phone, thereby realizing real-time and uninterrupted localization in the large-scale scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To describe the technical solutions in embodiments of the present disclosure or in the prior art more clearly, the accompanying drawings required in the embodiments are briefly described below. Apparently, the accompanying drawings in the following description show merely some embodiments of the present disclosure, and other drawings can be derived from these accompanying drawings by those of ordinary skill in the art without creative efforts.

[0038] FIG. 1 is a flowchart of an indoor 3D localization method;

[0039] FIG. 2 is a structural view7 of an indoor 3D localization system;

[0040] FIG. 3 is a schematic view' for constructing a 3D map model based on a smartphone,

[0041] FIG. 4 is a flowchart of loosely-coupled fusion based on visual and inertial information; and

[0042] FIG. 5 illustrates a specific application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The technical solutions of the embodiments of the present disclosure are clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are merely a part rather than all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0044] An objective of the present disclosure is to provide an indoor 3D localization method and system, to realize real-time and uninterrupted localization in a large-scale scene.

[0045] To make the above objectives, features, and advantages of the present disclosure clearer and more comprehensible, the present disclosure will be further described in detail below' with reference to the accompanying drawings and the specific examples.

[0046] As shown in FIG. 1, an indoor 3D localization method includes the following steps:

[0047] Step 101: Construct priori information of a localization region in an indoor scene, and construct a local 3D model of the localization region according to the priori information to form a visual localization priori database, the priori information including a point cloud location and 2D image features, and the visual localization priori database including 2D image features and 3D point cloud features.

[0048] A priori map is constructed. Upon determination of the localization region, scene videos are shot with a camera of a smartphone at a frame rate of 30 Hz. To ensure accurate model construction in the large-scale indoor scene, 3D models are reconstructed piecewise with image-based 3D reconstruction software such as Meshroom, Colmap and VisualSFM. Scales of the piecewise reconstructed 3D models are corrected by a total station. Small scale-corrected 3D models are spliced in combination with manual surveying and an ICP method. 2D image features and 3D point cloud features are stored to form the visual localization priori database.

[0049] Step 102: Acquire, in response to a localization request of a target person, a current scene image with a camera of a mobile phone, and determine a camera pose according to the current scene image and the visual localization priori database, the camera pose being a visual localization result.

[0050] During visual localization, the user sends the localization request. The camera is turned on to shoot a scene image at a current location. The smartphone uploads the current scene image to a cloud processor. The processor extracts 2D features of the current scene image, and associates and compares the 2D features with the 2D image features and the 3D point cloud features in the visual localization priori database. The processor determines reliable feature matching pairs between the 2D features of the current image and the 3D point cloud features with an RANSAC method, acquires the camera pose through a PnP algorithm on matching interior points, and transmits the camera pose to the user. [0051 ] Step 103: Acquire sensor data of an inertial component of a built-in MEMS of the mobile phone, and determine a relative localization result according to the sensor data, the sensor data including magnetic field intensity data, angular acceleration data, and acceleration data.

[0052] Step 103 specifically includes: Determine a stride length at a current moment according to acceleration data in one step of the target person; acquire a current location point of the target person, and determine a pose of the target person at the current moment according to the current location point and the sensor data; and determine the relative localization result of the target person according to the stride length at the current moment, the pose at the current moment and a location at a previous moment.

[0053] When the target person is a rescue worker, a pose of the rescue worker at the current moment is determined according to a recursive equation.

[0054] The recursive equation is expressed as:

[0055]

[0056] where, k is a location of the target person at a moment k, k is a speed of the target C - - v person at the moment k, is a pose matrix of the target person at the moment k, "'k4 is a location of the target person at a moment k-1, is a speed of the target person at the moment C”, fh k-1, is a pose matrix of the target person at the moment k-1, k is acceleration data of the target person at the moment k, ^k is angular acceleration data of the target person at the b b moment k, "is a zero offset of an accelerometer for acquiring the acceleration data, 2 is a zero offset of a gyroscope for acquiring the angular acceleration data, is a time interval, is a gravity vector of a navigation coordinate system, and &is an antisymmetric matrix.

[0057] Step 104: Determine whether the visual localization result is credible based on a visual localization credibility, execute Step 105 if yes, or execute Step 106 if no.

[0058] Step 105: Output the visual localization result.

[0059] Step 106: Output the relative localization result.

[0060] The present disclosure realizes accurate 3D localization of the pedestrian through the built-in camera and MEMS at the smartphone side. Without an additional signal transmitter, the present disclosure is applied to all indoor scenes. The present disclosure combines visual localization with inertial localization, and makes global localization and relative localization complementary to each other, thereby realizing decimeter-level localization accuracy in a 3D space. The present disclosure has desirable feasibility and applicability in the large-scale indoor scene, and improves a location-based sendee (LBS) experience of the user in the large-scale indoor scene.

[0061] As shown in FIG. 2, the present disclosure further provides an indoor 3D localization system, including: a visual localization priori database formation module, a camera pose determination module, a MEMS-based relative localization module, a determining module, a visual localization result output module, and a relative localization result output module.

[0062] The visual localization priori database formation module is configured to construct priori information of a localization region in an indoor scene, and construct a local 3D model of the localization region according to the priori information to form a visual localization priori database, the priori information including a point cloud location and 2D image features, and the visual localization priori database including 2D image features and 3D point cloud features.

[0063] The camera pose determination module is configured to acquire, in response to a localization request of a target person, a current scene image with a camera of a mobile phone, and determine a camera pose according to the current scene image and the visual localization priori database, the camera pose being a visual localization result.

[0064] As shown in FIG. 2, the visual global localization module includes the visual localization priori database formation module and the camera pose determination module. The 3D model of the localization region is constructed by the surveyor with a camera of a smartphone or a professional camera. The priori information of the localization region, including the point cloud location and the corresponding 2D image features, is constructed with a structure from motion (SFM) method. This process is to construct mutual correlation between image features and point cloud information of the localization region, namely 2D-3D data mapping. In response to the localization request of the user, the camera of the mobile phone is turned on to acquire image information of the scene, extract oriented fast and rotated brief (ORB) features of the image, and match features of the image with features of the local 3D model. To avoid wrong matching, the RANS AC algorithm is used to obtain accurate 2D-3D matching, and the number of iterations is set as 500. After a number of matching interior points reaches a threshold, internal parameters of the camera are obtained through an exchangeable image file format for storing the image. By solving an N~point PnP problem, a 6DoF pose of the camera is obtained.

[0065] The MEMS-based relative localization module is configured to acquire sensor data of an inertial component of a built-in MEMS of the mobile phone, and determine a relative localization result according to the sensor data, the sensor data including magnetic field intensity data, angular acceleration data, and acceleration data,

[0066] The 2D localization result is obtained by pedestrian dead reckoning (PDR), which includes estimation on a stride frequency, a stride length and a heading of the pedestrian. The PDR localization based on the built-in MEMS can quickly acquire planimetric location information of the user, and can provide a reliable 2D localization result in a short time.

[0067] The MEMS-based relative localization module acquires motion information of the user with an inertial component of the mobile phone, such as a gyroscope, an accelerometer and a magnetometer, and obtains the relative 3D location in combination with the PDR and an inertial navigation system (INS).

[0068] The MEMS-based relative localization module includes a gait detection unit, a stride length determination unit, a heading determination unit, and a location determination unit.

[0069] The gait detection unit is configured to determine a walking gait with peak detection and zero-cross detection according to the acceleration data of the target person.

[0070] The stride length determination unit is configured to determine a stride length at a current moment according to acceleration data in one step of the target person.

[0071] The heading determination unit is configured to determine a pose of the target person at the current moment according to a current location point of the target person as well as the magnetic field intensity data, the angular acceleration data and the acceleration data in the sensor data.

[0072] The location determination unit is configured to determine a location of the target person at the current moment, namely the relative localization result, according to the stride length and the pose of the target person at the cun-ent moment as well as a location at a previous moment.

[0073] When the target person is a rescue worker, the relative localization module includes a pose orchestration unit, a zero-speed detection unit, and a zero-speed correction unit,

[0074] The pose orchestration unit is configured to determine the pose of the target person at the current moment according to a recursive equation. The recursive equation is expressed as: 'A f' - h M\+v"Ar I I <- v”_, +| (At I v" vfI where, 'k is a location of the target person at a moment k, 4 is a speed of the target C" xn at the moment k, is a pose matrix of the target person at the moment k, w is a v” location of the target person at a moment k-1, w is a speed of the target person at the moment C", f' k-1, fiX !is a pose matrix ot the target person at the moment k-1, 4 is acceleration data of the

[0075]

[0076] person target person at the moment k, * is angular acceleration data of the target person at the b moment k, a is a zero offset of an accelerometer for acquiring the acceleration data, is a zero offset of a gyroscope for acquiring the angular acceleration data, is a time interval, is a gravity vector of a navigation coordinate system, and is an antisymmetric matrix. Generally, with the MEMS component in the smartphone, the localization result based on the relative localization module is obtained.

[0077] The determining module is configured to determine whether the visual localization result is credible based on a visual localization credibility.

[0078] The visual localization result output module is configured to output the visual localization result.

[0079] The relative localization result output module is configured to output the relative localization result.

[0080] The whole determining process is determined by a loosely-coupled fusion localization module. With estimation on the visual localization credibility, whether visual matching is failed is determined. The commonly-used number of interior points is used as a threshold indicator. When there are less than 12 interior points, the visual localization result is incredible, namely the relative localization result based on the MEMS is used, or otherwise, the visual localization result is used.

[0081] The indoor scene model is as shown in FIG. 3. During visual localization, the user sends the localization request. The camera is turned on to shoot a scene image at a current location. The smartphone uploads the image to a cloud processor. The processor extracts features of the image, and compares and associates the features with features in the database. The processor determines credible matching between the 2D features of the currently queried image and the 31) point cloud information with an RANSAC method, obtains a camera pose with a PnP algorithm on matching interior points, and transmits the camera pose to the user.

[0082] As shown in FIG. 4, loosely-coupled fusion is employed to ensure timeliness of the localization method and high localization efficiency of the LBS in the large-scale indoor scene. Specifically, the visual localization result is preferentially considered as an output of the localization method. Whether the visual localization is failed is determined through a number of interior points in feature matching of the visual localization. When the RANSAC method is used for 2D-3D matching, 500 iterations are performed. When there are less than 12 interior points, the matching is failed, the credible visual localization result cannot be obtained and the relative localization result based on the MEMS is selected as the output of the whole localization method. Or otherwise, the visual localization result is selected all the time.

[0083] As shown in FIG. 5, with a smartphone held by the user, the localization method provided by the present disclosure outputs a 3D location of the pedestrian in real time, and can provide cross-floor localization and navigation services for the user.

[0084] The present disclosure includes priori data acquisition and 3D model construction, MEMS-based relative localization, visual global localization, and vision / inertia loosely-coupled fusion localization.

[0085] The priori data acquisition is to acquire inertial data and image data, and perceive sensor data of the user at a current location. The 3D model construction is to shoot piecewise videos of a localization region with a camera of the smartphone, and obtain an accurate 3D model of the localization region in combination with an SFM method and a manual measurement of a total station.

[0086] The MEMS-based relative localization is to obtain a relative motion process of the user according to sensor data of the built-in MEMS to realize relative localization.

[0087] The visual global localization is to use image information to perceive image information of an environment where the user is located, query image features, match the image features with features in a priori map database, and solve a camera pose to obtain an accurate 3D location of the user.

[0088] The loosely-coupled fusion localization is to output a result of the whole localization method, thereby providing the real-time, accurate, reliable and continuous 3D localization result for the user.

[0089] Each embodiment of the present specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts between the embodiments may refer to each other. Since the system disclosed in an example corresponds to the method disclosed in another example, the description is relatively simple, and reference can be made to the method description.

[0090] Specific examples are used herein to explain the principles and implementations of the present disclosure. The foregoing description of the examples is merely intended to help understand the method of the present disclosure and its core ideas; besides, various modifications may be made by those of ordinary skill in the art to specific implementations and the scope of application in accordance with the ideas of the present disclosure. In conclusion, the content of the present specification shall not be construed as limitations to the present disclosure.

Claims

1. An indoor three-dimensional (3D) localization method, comprising:constructing priori information of a localization region in an indoor scene, and constructing a local 3D model of the localization region according to the priori information to form a visual localization priori database, the priori information comprising a point cloud location and two-dimensional (2D) image features, and the visual localization priori database comprising 2D image features and 3D point cloud features;acquiring, in response to a localization request of a target person, a current scene image with a camera of a mobile phone, and determining a camera pose according to the current scene image and the visual localization priori database, the camera pose being a visual localization result;acquiring sensor data of an inertial component of a built-in microelectromechanical system (MEMS) of the mobile phone, and determining a relative localization result according to the sensor data, the sensor data comprising magnetic field intensity data, angular acceleration data, and acceleration data;determining whether the visual localization result is credible based on a visual localization credibility'; andoutputting the visual localization result if yes, oroutputting the relative localization result if no.

2. The indoor 3D localization method according to claim 1, wherein the constructing a local 3D model of the localization region according to the priori information to form a visual localization priori database specifically comprises:reconstructing, for the large-scale indoor scene, small 3D models piecewise according to the priori information;correcting scales of the small 3D models through a total station, and splicing scale-corrected 3D models based on manual surveying and an iterative closest point (TCP) method to form the local 3D model of the localization region, andstoring the 2D image features and the 3D point cloud features according to the local 3D model to form the visual localization priori database.

3. The indoor 3D localization method according to claim 1, wherein the determining a camera pose according to the current scene image and the visual localization priori database specifically comprises:acquiring 2D features of the current scene image;comparing and associating the 2D features with the 3D point cloud features, and determining feature matching pairs between the 2D features and the 3D point cloud features with a random sample consensus (RANSAC) algorithm; andobtaining interior points of the feature matching pairs with a perspective-n-point (PnP) algorithm to determine the camera pose.

4. The indoor 3D localization method according to claim 1, wherein the determining a relative localization result according to the sensor data specifically comprises:determining a stride length at a current moment according to acceleration data in one step of the target person;acquiring a current location point of the target person, and determining a pose of the target person at the current moment according to the current location point and the sensor data; anddetermining the relative localization result of the target person according to the stride length at the current moment, the pose at the current moment and a location at a previous moment.

5. The indoor 3D localization method according to claim 4, wherein when the target person is a rescue worker, a pose of the rescue worker at the current moment is determined according to a recursive equation.

6. The indoor 3D localization method according to claim 5, wherein the recursive equation is expressed as:v                                                   v”wherein, -f is a location of the target person at a moment k, is a speed of the targetC” xr‘person at the moment k, 'h-k is a pose matrix of the target person at the moment k, ’ kA is alocation of the target person at a moment. k-L is a speed of the target person at. the momentC”, fhk-L is a pose matrix of the target person at the moment k-L k is acceleration data of the target person at the moment k, is angular acceleration data of the target person at theb bmoment k, 17 is a zero offset of an accelerometer for acquiring the acceleration data, g is azero offset of a gyroscope for acquiring the angular acceleration data, is a time interval. § is a gravity vector of a navigation coordinate system, and i$ an antisymmetric matrix.

7. An indoor three-dimensional (3D) localization system, comprising:a visual localization priori database formation module configured to construct priori information of a localization region in an indoor scene, and construct a local 3D model of the localization region according to the priori information to form a visual localization priori database, the priori information comprising a point cloud location and two-dimensional (2D) image features, and the visual localization priori database comprising 2D image features and 3D point cloud features;a camera pose detennination module configured to acquire, in response to a localization request of a target person, a current scene image with a camera of a mobile phone, and determine a camera pose according to the current scene image and the visual localization priori database, the camera pose being a visual localization result;a microelectromechanical system (MEMS)-based relative localization module configured to acquire sensor data of an inertial component of a built-in MEMS of the mobile phone, and determine a relative localization result according to the sensor data, the sensor data comprising magnetic field intensity data, angular acceleration data, and acceleration data,a determining module configured to determine whether the visual localization result is credible based on a visual localization credibility;a visual localization result output module configured to output the visual localization result; anda relative localization result output module configured to output the relative localization result.

Images