Indoor map matching method based on visual SLAM

Abstract

The invention provides an indoor map matching method based on visual SLAM (Simultaneous Localization and Mapping). The indoor map matching method comprises the following steps: S1, carrying out video data acquisition by using a 360-degree camera; s2, track key frame reconstruction in the walking process of the video photographer is carried out by utilizing a visual SLAM algorithm based on the video data; and S3, based on the plane drawing and the drawing calibration drawing of the video data area as constraint conditions, projecting the data of the spatial trajectory key frame from the three-dimensional space to the plane drawing and generating a trajectory video conforming to a shooting time sequence. The method can be widely applied to construction site inspection, the efficiency and quality of construction site inspection are improved through an offline analysis mode, and the method has extremely high application value and good application prospects.

Description

Technical Field

[0001] The invention relates to the field of image processing technology, and specifically to an indoor map matching method based on visual SLAM. Background Art

[0002] Nowadays, the demand for indoor map matching tasks in various social scenarios is increasing, especially in some public facilities, interior design and construction projects.

[0003] For map matching, location information is essential. The current positioning technologies mainly include RFID positioning technology, Bluetooth positioning technology, Wi-Fi positioning technology, UWB positioning technology, geomagnetic positioning technology, inertial sensor positioning technology, etc. These positioning technologies have high requirements for equipment. Among them, RFID positioning technology, Bluetooth positioning technology, Wi-Fi positioning technology, UWB positioning technology, etc. rely on on-site deployment of equipment, which is not only cumbersome, but also costly and requires professional operation. Summary of the invention

[0004] The purpose of the present invention is to provide an indoor map matching method based on visual SLAM to solve the problems of high cost, cumbersome deployment process and complex operation.

[0005] To achieve the above object, the present invention provides the following technical solution: The present invention provides an indoor map matching method based on visual SLAM, comprising the following steps:

[0006] S1, use 360 ​​camera to collect video data;

[0007] S2, based on the video data, the visual SLAM algorithm is used to reconstruct the trajectory key frames of the video shooter during his walking process;

[0008] S3. Based on the plane drawing and the drawing calibration map of the video data area as constraints, the data of the spatial trajectory key frame is projected from the three-dimensional space to the plane drawing and a trajectory video that conforms to the shooting time sequence is generated.

[0009] Preferably, S2 includes obtaining a three-dimensional point cloud image through a visual SLAM algorithm, obtaining a key frame of a trajectory through a visual SLAM algorithm, and obtaining a common frame of a trajectory through a visual SLAM algorithm.

[0010] Preferably, S3 includes:

[0011] S4, using the base plane drawing as a constraint condition, fitting a reasonable plane using the spatial trajectory key frame data, projecting the spatial trajectory key frame data onto the base plane based on the fitting plane, and generating trajectory key frame projection points;

[0012] S5. Based on the IOU values ​​of the calibration graph and the trajectory key frame projection points as constraints, a brute force search algorithm is used to perform matching to obtain the corresponding trajectory key frame projection points.

[0013] Preferably, the S4 includes:

[0014] S6. Based on the spatial trajectory key frame data, a general plane formula of the key frame data is fitted using the RANSAC algorithm and is represented by ax+by+cz+d=0;

[0015] S7, determining a reference plane having the smallest angle with the plane based on a general plane formula;

[0016] S8. Based on the normal vector of the general plane formula and the normal vector of the reference plane, the rotation formula of the general plane formula to the reference plane is obtained by using the method of inverse Rodriguez rotation formula;

[0017] S9. Based on the spatial trajectory key frame data, using the rotation formula, generate trajectory key frame projection points projected onto the reference plane, and connect the trajectory key frame projection points in time sequence to generate a trajectory projection.

[0018] Preferably, the S5 includes:

[0019] S10, based on the trajectory projection, using Hough transform to detect line segments in the graph;

[0020] S11. Based on the longest line segment among the line segments, obtain the length and radian corresponding to the line segment;

[0021] S12, based on the radian, rotating the trajectory projection to a direction perpendicular to the X-axis;

[0022] S13. Using a brute force search algorithm, change the four variables of rotation angle, scaling ratio, displacement in the X-axis direction, and displacement in the Y-axis direction. The rotation angle ranges from 0 to 360 degrees with a step length of 90 degrees. The scaling ratio ranges from 1 to 1.5 with a step length of 0.05. The displacements in the X-axis and Y-axis range from -300 to 350 pixels with a step length of 50 pixels.

[0023] The present invention has at least the following beneficial effects:

[0024] The present invention provides an indoor map matching method based on visual SLAM, which has low equipment requirements, can be met by consumer-grade products, and has low manual operation. The method can be widely used in construction site inspections, and can improve the efficiency and quality of construction site inspections by offline analysis. It has extremely high application value and good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the drawing structure of the indoor map matching method based on visual SLAM provided by the present invention;

[0026] Figure 2 This is a matching diagram of a trajectory projection diagram and a drawing of an indoor map matching method based on visual SLAM provided by the present invention. DETAILED DESCRIPTION

[0027] To facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that when an element is described as being "fixed to" another element, it can be directly on the other element, or one or more intermediate elements can exist therebetween; when an element is described as being "connected to" another element, it can be directly connected to the other element, or one or more intermediate elements can exist therebetween; the terms "upper", "lower", "inner", "outer", "bottom", etc. used in this specification to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention; in addition, the terms "first", "second", "third", etc., etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance.

[0028] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by technicians in the technical field to which the invention belongs. The terms used in this specification and in the description of the present invention are only for the purpose of describing specific embodiments and are not used to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0029] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] An indoor map matching method based on visual SLAM may include the following steps:

[0031] S1. Wear a hat equipped with a 360-degree camera and walk around the venue to collect video information along the way.

[0032] S2. Reconstructing the key frames of the trajectory of the video shooter during his walking process using a visual SLAM algorithm based on the video data. Specifically, reconstructing the three-dimensional point cloud map of the scene in the video and the ordinary frames and key frames of the walking trajectory using the visual SLAM algorithm;

[0033] S3. Based on the plane drawing and the drawing calibration map of the video data area as constraints, the data of the spatial trajectory key frame is projected from the three-dimensional space to the plane drawing and a trajectory video that conforms to the shooting time sequence is generated.

[0034] Among them, S3 includes:

[0035] S4, the base plane drawing is used as a constraint condition, a reasonable plane is fitted using the spatial trajectory key frame data, and the spatial trajectory key frame data is projected onto the base plane based on the fitted plane to generate trajectory key frame projection points;

[0036] S5. Based on the IOU values ​​of the calibration graph and the trajectory key frame projection points as constraints, a brute force search algorithm is used to match and obtain the corresponding trajectory key frame projection points.

[0037] Among them, S4 includes:

[0038] S6. Based on the spatial trajectory key frame data, the general plane formula of the key frame data is fitted using the RANSAC algorithm and is represented by ax+by+cz+d=0;

[0039] S7, determining the reference plane with the smallest angle to the plane based on the general plane formula;

[0040] S8. Based on the normal vector of the general plane formula and the normal vector of the reference plane, the rotation formula of the general plane formula to the reference plane is obtained by using the method of inverse Rodriguez rotation formula;

[0041] S9. Based on the spatial trajectory key frame data, using the rotation formula, generate trajectory key frame projection points projected onto the reference plane, and connect the trajectory key frame projection points in time sequence to generate trajectory projection.

[0042] Among them, S5 includes:

[0043] S10, based on trajectory projection, using Hough transform to detect line segments in the image;

[0044] S11. Based on the longest line segment among the line segments, obtain the length and arc corresponding to the line segment;

[0045] S12, based on the radian, rotating the trajectory projection to a direction perpendicular to the X-axis;

[0046] S13. Use a brute force search algorithm to change the four variables of rotation angle, scaling ratio, displacement in the X-axis direction, and displacement in the Y-axis direction. The rotation angle ranges from 0 to 360 degrees with a step size of 90 degrees. The scaling ratio ranges from 1 to 1.5 with a step size of 0.05. The displacement ranges of the X-axis and Y-axis are both -300 to 350 pixels with a step size of 50 pixels.

[0047] Specifically, the specific process of determining the reference plane includes:

[0048] The normal vector (a, b, c) of the general plane and the unit normal vectors (1, 0, 0), (0, 1, 0) and (0, 0, 1) of the three reference planes in space are calculated using the space vector angle calculation formula:

[0049]

[0050] in is the general plane normal vector, As the unit normal vector of the reference plane, the absolute value of COS is the maximum. The corresponding reference plane is identified as the plane with the smallest angle with the general plane.

[0051] Using the Rodriguez rotation formula, the rotation matrix from the general plane to the reference plane with the minimum angle is calculated in reverse.

[0052] Furthermore, the specific process of finding the rotation matrix includes:

[0053]

[0054] s=||v||

[0055]

[0056] in is the general plane normal vector, is the unit normal vector of the reference plane, and v is The normal vector of the plane, s is the bi-norm of v, and c is and The dot product of

[0057] The calculation formula for solving the rotation matrix is:

[0058]

[0059] Among them, [v] x is a skew-symmetric matrix of v, in matrix form:

[0060]

[0061] Based on the rotation matrix, the trajectory data is rotated and projected onto the determined reference plane;

[0062] Furthermore, the projection process includes:

[0063] According to the determined reference plane, the data in the corresponding two coordinate axes are selected as projection data. For example, if the determined reference plane is the XOZ plane, all Y-direction data in the rotated trajectory data are set to zero;

[0064] According to the processed projection data, the x-direction and y-direction data of the drawing in the canvas are used in sequence. For example, when the reference plane is the XOZ plane, the x-direction data in the rotated trajectory data is used as the x-direction data in the canvas, and the z-direction data is used as the y-direction data in the canvas.

[0065] Based on the drawing, calibrate the length and width of the drawing as the canvas size;

[0066] Since the original canvas trajectory data is in meters and the value is very small, direct drawing will result in failure to present a reasonable size in the canvas in pixels. Therefore, the original canvas trajectory data needs to be scaled. Based on the trajectory position data in the canvas, the scaling ratio of the original canvas trajectory data is calculated. The scaling ratio calculation formula is:

[0067] r x =(m w -300)÷(t xmax -t xmin )

[0068] r y =(m h -300)÷(t ymax -t ymin )

[0069] Where m w and m h are the pixel values ​​of the width and height of the calibration image, t xmax and t xmin is the maximum and minimum value of the x direction in the canvas trajectory data, t ymax and t ymin is the maximum and minimum value in the x direction of the canvas trajectory data, and finally in r x and r y The minimum value of the two scaling ratios for the x direction and the y direction is selected as the scaling ratio of the original canvas trajectory data;

[0070] Since the original data has positive and negative values, directly drawing the track on the canvas will result in incomplete or even no track on the canvas. Therefore, it is necessary to shift the scaled track data. The shift operation includes:

[0071] To calculate the displacement of the data as a whole in the x and y directions, simply add it to the x and y data in the data. The displacement calculation formula is:

[0072] move x =t xmax × x -((t xmax -t xmin )÷2)×r x

[0073] move y =t ymax × y -((t ymax -t ymin )÷2)×r y

[0074] move X =m w ÷2-move x

[0075] move Y =m h ÷2-move y

[0076] Among them move x and move Y are the displacements in the x and y directions respectively.

[0077] Based on the canvas trajectory data after scaling and displacement, draw it on the canvas;

[0078] Since the direction of drawing the trajectory graph is not parallel to the x-direction or the y-direction, this will increase the rotation angle that the brute force search algorithm needs to traverse. In order to reduce the complexity of the brute force search, a rotation operation is performed on the drawing trajectory graph. The rotation operation includes:

[0079] Based on drawing the trajectory graph, a dilation operation with a dilation kernel of 7 is used;

[0080] The Hough transform detection function is used to detect the line segments in the trajectory map after expansion. The arc of the longest line segment is used as the rotation angle to reversely rotate the trajectory map after expansion, and finally generate Figure 2 The trajectory diagram shown.

[0081] Based on the trajectory graph, a brute force search algorithm is used to change the four variables of rotation angle, scaling ratio, displacement in the X-axis direction, and displacement in the Y-axis direction. The rotation angle ranges from 0 to 360 degrees with a step size of 90 degrees, the scaling ratio ranges from 1 to 1.5 with a step size of 0.05, and the displacement ranges of the X-axis and Y-axis are both -300 to 350 pixels with a step size of 50 pixels.

[0082] Based on the brute force search algorithm, during the operation of the algorithm, the IOU value of the trajectory map after each scaling, translation and rotation is calculated with the calibration map of the drawing. The IOU calculation formula is:

[0083]

[0084] Pred is the transformed trajectory map, and GT is the calibration map.

[0085] Since the trajectory should be located in the corridor in the calibration diagram, and the highlighted part in the calibration diagram is the wall part, the trajectory and the drawing are most matched only when the IOU value between the trajectory and the highlighted part is the smallest. Therefore, when the minimum value appears after comparing the two, the corresponding scaling ratio, displacement, and rotation angle are retained;

[0086] Based on the scaling, displacement, and rotation angle, the trajectory diagram after the initial transformation is transformed and drawn in sequence based on the time axis in the original trajectory keyframe data. Figure 2 shown.

[0087] This method can be widely used in construction site inspections. It can improve the efficiency and quality of construction site inspections through offline analysis. It has extremely high application value and good application prospects.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for indoor map matching based on visual SLAM, characterized in that: The following steps are involved: S1, use 360 ​​camera to collect video data; S2, based on the video data, the visual SLAM algorithm is used to reconstruct the trajectory key frames of the video shooter during his walking process; S3. Based on the plane drawing and the drawing calibration map of the video data area as constraints, the data of the spatial trajectory key frame is projected from the three-dimensional space to the plane drawing and a trajectory video that conforms to the shooting time sequence is generated.

2. The indoor map matching method based on visual SLAM according to claim 1, characterized in that: The S2 includes obtaining a three-dimensional point cloud image through a visual SLAM algorithm, obtaining a key frame of a trajectory through a visual SLAM algorithm, and obtaining a common frame of a trajectory through a visual SLAM algorithm.

3. The indoor map matching method based on visual SLAM according to claim 1, characterized in that: The S3 includes: S4, using the base plane drawing as a constraint condition, fitting a reasonable plane using the spatial trajectory key frame data, projecting the spatial trajectory key frame data onto the base plane based on the fitting plane, and generating trajectory key frame projection points; S5. Based on the IOU values ​​of the calibration graph and the trajectory key frame projection points as constraints, a brute force search algorithm is used to perform matching to obtain the corresponding trajectory key frame projection points.

4. The indoor map matching method based on visual SLAM according to claim 3 is characterized in that: The S4 includes: S6. Based on the spatial trajectory key frame data, a general plane formula of the key frame data is fitted using the RANSAC algorithm and is represented by ax+by+cz+d=0; S7, determining a reference plane having the smallest angle with the plane based on a general plane formula; S8. Based on the normal vector of the general plane formula and the normal vector of the reference plane, the rotation formula of the general plane formula to the reference plane is obtained by using the method of inverse Rodriguez rotation formula; S9. Based on the spatial trajectory key frame data, using the rotation formula, generate trajectory key frame projection points projected onto the reference plane, and connect the trajectory key frame projection points in time sequence to generate a trajectory projection.

5. The indoor map matching method based on visual SLAM according to claim 3 is characterized in that: The S5 includes: S10, based on the trajectory projection, using Hough transform to detect line segments in the graph; S11. Based on the longest line segment among the line segments, obtain the length and radian corresponding to the line segment; S12, based on the radian, rotating the trajectory projection to a direction perpendicular to the X-axis; S13. Using a brute force search algorithm, change the four variables of rotation angle, scaling ratio, displacement in the X-axis direction, and displacement in the Y-axis direction. The rotation angle ranges from 0 to 360 degrees with a step length of 90 degrees. The scaling ratio ranges from 1 to 1.5 with a step length of 0.

05. The displacements in the X-axis and Y-axis range from -300 to 350 pixels with a step length of 50 pixels.

Images