A monocular camera-based prior map assisted indoor positioning method

Abstract

The application provides a monocular camera-based prior map-assisted indoor positioning method. The method comprises the following steps: S1, acquiring a prior point cloud map of a non-closed-loop indoor scene; S2, performing semantic segmentation on the known prior map, extracting local point clouds retaining environmental semantic information, and forming a simplified semantic line feature positioning map; S3, extracting two-dimensional line features from a camera image, and adding a semantic detection process to the two-dimensional straight lines and adding semantic labels to the two-dimensional straight lines; S4, for a single frame image, judging the visibility of line segments in the semantic positioning map, matching the two-dimensional line features with three-dimensional line segments in the field of view and having consistent semantic labels; and S5, constructing an optimization model matched with the prior map in real time, and improving the positioning accuracy of the system. The method solves the problem that it is difficult to rely on closed-loop correction cumulative error as a pose optimization basis in a large-scale indoor scene, and effectively improves the accuracy of the large-scale indoor positioning result of the monocular visual positioning system.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a monocular camera-based prior map-assisted indoor positioning method. Background Art

[0002] Among the many methods of indoor positioning, SLAM visual positioning plays an increasingly important role. As we all know, SLAM technology generally relies on closed-loop correction to correct accumulated errors. However, in large indoor scenes such as airports, shopping malls, and museums, closed loops may not exist for a long time. Therefore, achieving continuous and robust positioning of large-scale indoor visual SLAM requires the use of other pose optimization methods.

[0003] Indoor environments contain a large amount of environmental semantic information, such as doors and windows, as well as clear, structured line features. Line features are more descriptive than point features in the same scene. In theory, given a relatively accurate 3D environmental map, by matching the SLAM-reconstructed environment with existing map model data (such as a BIM model) in real time, continuous, high-precision positioning can be achieved in large indoor scenes without closed loops. Summary of the Invention

[0004] In order to solve the above-mentioned problem that it is difficult to rely on closed-loop correction cumulative error as the basis for pose optimization in large-scale indoor scenes, the present invention proposes a priori map-assisted indoor positioning method based on a monocular camera, which can effectively improve the accuracy of large-scale indoor positioning results.

[0005] To achieve the above objectives, the present invention provides a method for indoor positioning assisted by a priori maps based on a monocular camera, comprising the following steps:

[0006] S1 obtains the prior point cloud map of the indoor scene without closed loop;

[0007] S2 performs semantic segmentation on the known prior map, extracts local point clouds with environmental semantic information, and extracts three-dimensional line segments to form a simplified semantic line feature positioning map;

[0008] S3 extracts 2D line features from camera images, adds a semantic detection process, adds semantic labels to 2D line segments, and performs filtering and merging.

[0009] S4 determines the visibility of the 3D line segments in the semantic localization map for a single frame image, and matches the 2D line features with the 3D line segments within the field of view and with the same semantic labels;

[0010] S5 builds an optimized model for matching real-time maps and prior maps to improve system positioning accuracy.

[0011] S6 compares and tests the performance of each algorithm under non-closed trajectory and circular trajectory, using absolute position error and closure error as the evaluation criteria for the performance of each algorithm.

[0012] In step 1, a handheld Kinect-v2.0 camera is used to capture the RGB image and depth map of the scene, and ORB-SLAM2 is used to generate dense point cloud information, which is stored as a priori point cloud map.

[0013] In step 2, the efficient semantic segmentation model RandLA-Net is used to significantly reduce the point density using a simple and fast random sampling method, while a redesigned LocalFeature aggregator is applied to retain significant features.

[0014] Among them, in step 2, in order to meet the requirements of three-dimensional straight line extraction and obtain better robustness, an image-based three-dimensional line segment detection method is used to extract three-dimensional straight lines, and combined with the semantic segmentation model RandLA-Net, a priori map consisting of line segments with environmental semantic information is effectively generated.

[0015] In step 3, the M-LSD algorithm is used for 2D line feature detection. A semantic detection thread is also added, and semantic segmentation is performed using a SegNet network. If the extracted line segment is within the color block corresponding to the door, the corresponding semantic category label is assigned. Line segments labeled "door" are filtered, and the line group close to the semantic boundary curve ξ is retained. 2D line segments that may be the same door frame line are merged.

[0016] Wherein, in said step 4, depending on the visibility of the endpoints of the door frame line, different strategies are adopted in three cases to judge the visibility of the three-dimensional door frame line.

[0017] Among them, in the step 4, after screening, the two end points of the door frame line L are projected into the image plane respectively, and all three-dimensional door frame lines within the field of view are traversed. The angle, length difference and distance between the door frame lines and the two-dimensional door frame lines are calculated, and the two-dimensional-three-dimensional matching pairs that meet the three parameters that are all less than the threshold are retained to complete the matching with the prior map.

[0018] In step 5, the motion between two consecutive frames is iteratively estimated by using the Ceres Solver solver, and a matching optimization model is constructed to improve the positioning accuracy of the system.

[0019] Beneficial effects of the present invention:

[0020] This method first performs semantic segmentation on a priori maps to extract semantically relevant line features from the map, creating a streamlined semantic line feature positioning map. Then, during actual positioning, it extracts and processes line features from a monocular camera online, filters out doorframe lines, and matches them with the pre-stored map. A system-optimized cost function is constructed to effectively optimize the camera pose without a closed loop. Finally, the algorithm's performance is validated using a real-world dataset. Experiments demonstrate that this method effectively improves the accuracy of large-scale indoor positioning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 , is a flow chart of a method for indoor positioning assisted by a priori map based on a monocular camera in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inward" and "outward" refer to directions toward or away from the geometric center of a particular component, respectively.

[0023] Figure 1 This is a flow chart of a method for indoor positioning assisted by a priori maps based on a monocular camera in an embodiment of the present invention;

[0024] like Figure 1 As shown, the present invention provides a monocular camera-based prior map-assisted indoor positioning method, which specifically includes the following steps:

[0025] S1 obtains the prior point cloud map of the indoor scene without closed loop;

[0026] Specifically, the prior map data for this example is collected from the RGB images and depth maps of the scene captured by the Kinect-v2.0 camera. The user walks in the corridor environment where the experiment is located with a handheld Kinect-v2.0 camera to obtain scene information. The dense point cloud is calculated by ORB-SLAM2. After running the ORB-SLAM2 algorithm, the dense point cloud information is stored in .ply format as the prior map for subsequent experiments.

[0027] S2 performs semantic segmentation on the known prior map, extracts local point clouds with environmental semantic information, and extracts three-dimensional line segments to form a simplified semantic line feature positioning map;

[0028] Specifically, we utilize the efficient semantic segmentation model RandLA-Net, use a simple and fast random sampling method to significantly reduce the point density, and apply a redesigned LocalFeature aggregator to retain significant features.

[0029] S3 extracts 2D line features from camera images, adds a semantic detection process, adds semantic labels to 2D line segments, and performs filtering and merging.

[0030] Specifically, the M-LSD algorithm is used to detect 2D line features on the RGB image, and the segnet network is used for semantic segmentation. The two threads process the RGB image simultaneously. Subsequently, the extracted 2D line features are given corresponding semantic category labels, and all line segments labeled "door" are filtered and those with length less than the threshold l are removed. d and the line segment far from the boundary of the semantic category. Calculate the distance dist(p, ξ) between the two endpoints and the midpoint of the line segment and the semantic boundary ξ, discard the maximum value and add the two smaller values, denoted as ∑ i When Σ i Less than the threshold Σ d When , it is considered that the line segment belongs to the searched two-dimensional door frame line V d .

[0031] Finally, we have V d If the overlapping part O of the line segments in the X-axis or Y-axis direction is greater than the threshold O d , then determine the angles α and β between the two and the coordinate axis. If |α-β| is less than the threshold γ d It is considered that the two can be merged, and the two farther endpoints are taken to represent the new line segment V.

[0032] S4 determines the visibility of the required type of line segments in the 3D map for a single frame image, and matches the 2D line features with the same semantic labels with the 3D line segments retained within the field of view;

[0033] Specifically, the 3D door frame lines are filtered based on the visibility of the endpoints, and line segments that are not within the field of view in the current frame are removed. The endpoints of the door frame lines are extracted and processed using different strategies in three different situations:

[0034] (1) If both end points are within the field of view, then the door frame line is also considered to be within the field of view;

[0035] (2) One of the two endpoints is within the field of view, and the other is outside the field of view. Keep the point X within the field of view. i . Take the midpoint of the two If it is within the field of view, keep the line segment If it is not within the field of view, the process is repeated for the newly generated line segment until the length of the line segment is less than the set threshold.

[0036] (3) If both end points are outside the field of view, the door frame line is considered to be outside the field of view.

[0037] Project the two endpoints of the 3D door frame line L onto the image plane respectively, traverse all 3D door frame lines within the field of view, calculate the angle θ, length difference Δl and distance d between them and the 2D door frame line, find the 2D-3D matching pair that satisfies θ<θ0, Δl<Δl0, d<d0, and complete the matching with the prior map.

[0038] S5 builds an optimized model for matching real-time maps and prior maps to improve system positioning accuracy.

[0039] Specifically, the error terms for matching two-dimensional line segments with three-dimensional line segments are defined. Under the constraints of the prior point cloud map, all error terms are added in the form of least squares to construct a cost function for solution.

[0040] For the constraints of the door edge in the prior map, we define the two endpoints after the three-dimensional semantic line feature is projected onto the two-dimensional plane The sum of the distances to the straight line Ax+By+C=0 where the 2D line segment is located is the new error term Then for a single frame image, the kth camera frame c k The observed i-th spatial line L i The residual is:

[0041]

[0042]

[0043]

[0044] The final cost function can be expressed as:

[0045]

[0046] Among them, the first term is the Marg marginalization residual part, the second term is the IMU residual part between adjacent frames in the sliding window, and the third term is the visual reprojection residual of the feature points in the sliding window under the camera. χ is all the state quantities in the sliding window, is the covariance matrix of the IMU pre-integrated noise term, is the covariance matrix of the visual feature point observation noise, is the covariance matrix of the observation noise of the 2D-3D semantic line feature.

[0047] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above-mentioned embodiment, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A priori map-assisted indoor positioning method based on a monocular camera, characterized in that: The following steps are involved: S1 obtains the prior point cloud map of the indoor scene without closed loop; S2 performs semantic segmentation on the known prior map, extracts local point clouds with environmental semantic information, and extracts three-dimensional line segments to form a simplified semantic line feature positioning map; S3 extracts 2D line features from the camera image, adds a semantic detection process, adds semantic labels to the 2D line segments, and filters and merges them; uses the M-LSD algorithm to detect 2D line features on the RGB image, and uses the segnet network to perform semantic segmentation. The two threads process the RGB image simultaneously; then, the extracted 2D line features are given corresponding semantic category labels, and all line segments labeled "door" are filtered to remove those with a length less than the threshold l d and the line segment far away from the boundary of the semantic category; calculate the distance dist(p, ξ) between the two end points and the midpoint of the line segment and the semantic boundary ξ respectively, discard the maximum value and add the two smaller values, denoted as ∑ i ; When Σ i Less than the threshold Σ d When , it is considered that the line segment belongs to the searched two-dimensional door frame line V d ; Finally, we have V d If the overlapping part O of the line segments in the X-axis or Y-axis direction is greater than the threshold O d , then determine the angles α and β between the two and the coordinate axis. If |α-β| is less than the threshold γ d It is considered that the two can be merged, and the two farther endpoints are taken to represent the new line segment V; S4 determines the visibility of line segments in the semantic localization map for a single frame image and matches the two-dimensional line features with three-dimensional line segments within the field of view and with the same semantic labels; S5 builds an optimization model for matching the real-time map and the prior map to improve the positioning accuracy of the system. In step S5, the motion between two consecutive frames is iteratively estimated by the Ceres Solver solver to build a matching optimization model to improve the positioning accuracy of the system, as follows; The process of minimizing the error model is the process of finding the optimal solution to the objective function in the error model; the objective function is expressed as follows; Among them, the first term is the Marg marginalization residual part, the second term is the IMU residual part between adjacent frames in the sliding window, and the third term is the visual reprojection residual of the feature points in the sliding window under the camera; χ is all the state quantities in the sliding window, is the covariance matrix of the IMU pre-integrated noise term, is the covariance matrix of the visual feature point observation noise, is the covariance matrix of the observation noise of the 2D-3D semantic line feature; Error term corresponding to 2D-3D matching pairs It can be expressed as: The equation of the straight line where the two-dimensional door frame line is located is Ax+By+C=0. and They represent the endpoint coordinates of the prior three-dimensional door frame line after being projected onto the image plane.

2. The method for indoor positioning based on a priori map assisted by a monocular camera according to claim 1, characterized in that: The following steps are involved: In step S2, the efficient semantic segmentation model RandLA-Net is used to significantly reduce the point density using a simple and fast random sampling method, while a redesigned LocalFeature aggregator is applied to retain significant features.

3. The method for indoor positioning based on a priori map assisted by a monocular camera according to claim 1, characterized in that: In step S3, in order to meet the requirements of three-dimensional straight line extraction and obtain better robustness, an image-based three-dimensional line segment detection method is used to extract line features.

4. The method for indoor positioning based on a priori map assisted by a monocular camera according to claim 1, characterized in that: In step S3, the M-LSD algorithm is used to perform two-dimensional line feature detection, and a semantic detection thread is added to add semantic labels to the two-dimensional line segments in the image. The two threads process the RGB image simultaneously.

5. The method for indoor positioning based on a priori map assisted by a monocular camera according to claim 1, characterized in that: In step S4, depending on the visibility of the door frame line endpoints, different strategies are used to screen the 3D door frame lines in three different situations. Specifically, line segments that are not within the field of view in the current frame are removed. The endpoints of the door frame lines are extracted, and different strategies are used in three different situations: (1) If both end points are within the field of view, then the door frame line is also considered to be within the field of view; (2) One of the two endpoints is within the field of view, and the other is outside the field of view; the point X that remains within the field of view i ; Take the midpoint of the two If it is within the field of view, keep the line segment If it is not within the field of view, the process is repeated for the newly generated line segment until the length of the line segment is less than the set threshold; (3) If both end points are outside the field of view, the door frame line is considered to be outside the field of view.

6. The method for indoor positioning assisted by a priori map based on a monocular camera according to claim 5, characterized in that: In step S4, after screening, the two endpoints of the door frame line L are projected onto the image plane respectively, and all three-dimensional door frame lines within the field of view are traversed. The angle θ, the length difference Δl, and the distance d between the door frame line and the two-dimensional door frame line are calculated, and a two-dimensional-three-dimensional matching pair that satisfies θ<θ0, Δl<Δl0, and d<d0 is found; thus, matching with the prior map is completed.

Images