Indoor map automatic construction method fusing public source track and mobile phone image

Abstract

The invention, which relates to the technical field of indoor map construction, discloses an automatic indoor map construction method integrating a crowd source track and a mobile phone image, comprising the following steps: S1, acquiring corridor, indoor images and multi-sensor data by using a smart phone; s2, constructing an indoor navigation map; s3, extracting an indoor plane structure; and S4, fusing and optimizing the navigation path and the indoor plane structure. According to the indoor map automatic construction method fusing the public source track and the mobile phone image, in the public source track collection process, the indoor image collection method is guided in a standardized mode, plane structure features in the image are extracted, and the space information of the map is further enriched. Overall and local advantages and framework and detail advantages of the two are complemented, so that on one hand, the inefficiency of crowd source data acquisition is effectively relieved; and on the other hand, a low-cost data acquisition mode is continued, the data quality is ensured, and the drawing precision is improved, so that the availability of the location service is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor map construction, and in particular to an automatic indoor map construction method integrating multi-source trajectories and mobile phone images. Background Art

[0002] With the rapid advancement of urbanization, indoor space is expanding. Research on indoor space information applications has attracted more and more attention from academia and industry, and the market size of indoor applications with location services as the core is growing rapidly. As the data basis for indoor location applications, indoor two-dimensional maps have problems such as lack of data, low mapping accuracy and efficiency due to the complex signal environment, spatial layout, and topology volatility, which have become the main factors restricting the development of indoor location services (LBS). Traditional indoor two-dimensional plane map construction can be divided into two methods based on manual mapping and professional sensors according to the data collection method. The manual mapping method uses tape measures, steel rulers, handheld laser rangefinders, total stations, etc. as basic measuring instruments, supplemented by mapping software such as AutoCAD for construction. Although it can effectively ensure map accuracy, manual modeling requires a lot of time and effort. The method based on professional sensors uses laser scanners or cameras to collect indoor space information to generate indoor two-dimensional maps. However, professional mapping equipment has high requirements for collection personnel and high hardware costs, making it difficult to popularize and promote in applications.

[0003] At present, the hardware of smartphones is changing very fast. They are equipped with multiple sensors (accelerometer, gyroscope, magnetometer, barometer, GPS, etc.), which not only enable them to have strong computing power, but also to obtain multi-source data (indoor movement trajectory, behavior data, images, etc.), making them an ideal platform for indoor mapping. Domestic and foreign scholars have explored a variety of indoor mapping methods based on different data sources. Among them, SLAM-based

[0004] The indoor mapping method of smartphones based on Simultanous Localization and Mapping (SLAM) has gradually matured, and scholars have also conducted a lot of explorations, which has promoted the development of SLAM technology in terms of mapping accuracy, scale, reliability, etc. However, this type of method requires users to always open the camera and aim it at the surrounding environment, which is not in line with the user's usage habits. In addition, the algorithm is heavily dependent on the image feature points in the scene, mismatching is widespread, and the memory and computing power of the mobile phone are required to be high during use. In addition, the indoor mapping method of smartphones based on crowd-source data effectively reduces the cost and technical threshold of data collection, but lacks effective management of the time, efficiency and quality of data collection. Due to the non-professional nature of data collection, it is difficult to ensure the accuracy of map construction, and a lot of energy is required for data screening and effective information extraction. Summary of the invention

[0005] The purpose of the present invention is to provide a method for automatically constructing indoor maps by integrating multi-source trajectories and mobile phone images. By utilizing the multi-source trajectory data collected by a smart phone platform and collecting indoor images according to the established shooting rules, accurate construction of indoor maps can be achieved, providing a low-cost, high-precision and high-efficiency composition method for indoor plane map construction.

[0006] To achieve the above object, the present invention provides a method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images, comprising the following steps:

[0007] S1, using smartphones to collect corridor, indoor images and multi-sensor data;

[0008] S2, build indoor navigation map;

[0009] S3, extracting indoor plane structure;

[0010] S4. Integrate and optimize the navigation path and indoor plane structure.

[0011] Preferably, S2 comprises the following steps:

[0012] S21, obtaining multi-source trajectory data by using pedestrian dead reckoning to collect multi-sensor data from the smartphone platform;

[0013] S22, obtaining behavior landmarks through a behavior recognition algorithm, and clustering the behavior landmarks through a clustering algorithm based on Wi-Fi fingerprints;

[0014] S23. Calculate the relative distances between the behavior landmarks to obtain an indoor navigation map.

[0015] Preferably, S3 comprises the following steps:

[0016] S31, inputting a monocular image of an indoor space captured by a smartphone, and extracting a three-dimensional layout structure of the monocular image through an indoor three-dimensional scene understanding network model;

[0017] S32, mapping the indoor 3D scene into the 2D image and performing projection transformation;

[0018] S33. Restore the spatial structure ratio and obtain the indoor plane structure.

[0019] Preferably, S32 comprises the following steps:

[0020] S321, using VPBC technology to extract and solve camera parameters based on vanishing point. The camera principal point is located at the center of the image. The camera intrinsic parameter matrix K is in the form of:

[0021]

[0022] f is the focal length, which can be found from the vanishing point;

[0023] S322, set the vanishing point in the image coordinate system to be vp0(x i0 ,y i0 ), vp1(x i1 ,y i1 ), vp2(x i2 ,y i2 ), since the centroid of the triangle formed by the three vanishing points is the intersection point p(ox,oy) of the camera optical axis and the imaging plane, the coordinates of vp0 in the camera coordinate system are (x c0 ,y c0 ,f), where x c0 =x i0 -ox,y c0 =-(y i0 -oy), the coordinates of the other two vanishing points can be obtained in the same way. Using the orthogonal relationship between the vanishing points, the focal length and rotation matrix can be obtained as follows:

[0024]

[0025] S323, using camera parameters to calculate the projection transformation matrix H ij , you can use the vanishing point vp i and vp j The determined plane is corrected by “orthographic projection”, H ij The solution is as follows:

[0026] H ij =K·R ij ·K -1 ;

[0027] Using the transformation matrix H of the front wall and floor 01 and H02 Reproject the 4 intersection points to get the coordinates of each point in the front view after eliminating distortion

[0028]

[0029] Preferably, the actual ratio of length, width and height of the indoor space is:

[0030]

[0031] Preferably, S4 comprises the following steps:

[0032] S41, corridor construction, matching the corrected trajectory route with the corridor plane structure extracted from the mobile phone image to realize map construction of the corridor area;

[0033] S42, matching the room plan structure extracted from the image with the inertial navigation data to achieve drawing of the room structure in the indoor map.

[0034] Preferably, the map accuracy is further optimized by constructing an energy function, and the energy equation is proposed to be constructed as follows:

[0035] min x ={x f ,R,t}E l (x)+E c (x)+E b (x);

[0036] Where E l (x) represents the complexity of the layout, E c (x) represents the degree of closure, E b (x) represents the boundary similarity between adjacent segments, {x f ,R,t} represents the rotation and displacement of all corridor paths and room layouts when they are connected.

[0037] Preferably, the conditions that restrict the uniqueness of the corridor and room layouts are:

[0038]

[0039] Preferably, the following rules are followed during image acquisition: when shooting indoor corridors, it is necessary to take pictures of all the fields of view facing the turning points in a clockwise order during the turning process; when shooting indoor rooms, it is necessary to shoot from both sides of the room toward the opposite side.

[0040] Therefore, the present invention adopts the above-mentioned method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images, which has the following beneficial effects:

[0041] (1) A high-efficiency, low-cost, and independent method for extracting the planar structure of indoor map elements is proposed. The method uses pictures taken by mobile phones as the data source. It does not require reconstruction of the geometric structure of indoor three-dimensional models, nor does it require professional measurement equipment. It only uses a smartphone to take monocular pictures of indoor scenes. Through visual scene understanding technology, the planar structure of indoor map elements can be obtained, effectively improving the authenticity and accuracy of indoor map expression.

[0042] (2) Different types of data collected only by smartphones are integrated with multi-source trajectories and standardized mobile phone image collection, and a new geometric feature extraction method is used to achieve high-precision construction of indoor maps. The advantages of indoor space features of overall and local, framework and details are fully utilized to complement each other. Users do not need to walk through all locations in the indoor environment, which effectively saves the labor cost of indoor map drawing and reduces time and calculation.

[0043] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the present invention;

[0045] Figure 2 This is a schematic diagram of relative distance estimation according to the present invention;

[0046] Figure 3 The step detection and angle measurement results between the behavioral landmarks of the present invention;

[0047] Figure 4 The corridor image acquisition rule of the present invention, a is a right-angle turn; b is a four-pronged corner; c is a T-shaped corner;

[0048] Figure 5 This is the room image acquisition rule of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0051] Example

[0052] See also Figure 1-5 The present invention provides a method for automatically constructing indoor maps by integrating multi-source trajectories and mobile phone images. Through the three modules of indoor navigation map construction, indoor plane structure extraction, and indoor map construction integrating navigation paths and plane structures, it only needs to use the multi-source trajectory data collected by the smart phone platform and collect indoor images according to the established shooting rules to realize the accurate construction of indoor maps, thereby providing a low-cost, high-precision and high-efficiency composition method for indoor plane map construction.

[0053] A method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images, comprising the following steps:

[0054] S1. Collect corridor, indoor images and multi-sensor data using smartphones.

[0055] S2. Build an indoor navigation map.

[0056] S21. Obtain crowdsourced trajectory data from the multi-sensor data collected by the smartphone platform through pedestrian dead reckoning. There are many behavioral landmarks in the crowdsourced data, some of which are collected at the same node. In order to construct an indoor map, the behavioral landmarks collected at the same node need to be clustered first.

[0057] S22. Obtain behavior landmarks through a behavior recognition algorithm, and cluster the behavior landmarks through a clustering algorithm based on Wi-Fi fingerprints.

[0058] When a new trajectory is obtained, the behavior recognition algorithm is used to obtain the behavior landmark, and the Wi-Fi fingerprint at the time of the behavior is extracted as the feature of the behavior landmark. A behavior landmark sequence {NAL1, NAL2...NAL m}. Among them, there are m behavioral landmarks (NAL represents the behavioral landmark extracted from the newly uploaded trajectory). The m behavioral landmarks are clustered using a clustering algorithm based on Wi-Fi fingerprints, among which there may be n behavioral landmarks with the same Wi-Fi fingerprint features, where (0≤n≤m).

[0059] If the trajectory is the first trajectory, the behavioral landmarks contained in the trajectory are added to the node database in sequence. If the trajectory is not the first trajectory data, a large number of existing garbage image datasets can be collected in the trajectory through a clustering algorithm based on Wi-Fi features to construct a training dataset.

[0060] S23. Calculate the relative distances between the behavior landmarks to obtain an indoor navigation map.

[0061] After the behavioral landmarks are clustered, all behavioral landmarks are clustered into different classes, each class represents a node in the indoor map, and if two nodes are directly adjacent, the distance between them can be obtained by pedestrian dead reckoning. Figure 4 As shown in (a), the figure shows a trajectory collected by a smartphone (obtained through pedestrian dead reckoning), and A, B, C, and D are four behavioral landmarks (turns). Among them, AB, BC, and CD are adjacent behavioral landmarks (known by the time sequence of landmark detection), and the relative distance between them can be directly obtained through step detection and step length estimation. For non-adjacent landmarks, such as A and C, the lengths of AB and BC and the angle information of angle B are required to calculate the relative distance between the landmarks. Figure 4 The distance and angle information between the behavioral landmarks in (a) can be obtained through the inertial data obtained by the smartphone. The distance information can be obtained by step detection through accelerometer data, while the angle information is obtained through the gyroscope and electronic compass. Figure 3 The step detection results and angle change information between different behavioral landmarks are displayed.

[0062] Based on the clustering of behavioral landmarks and the calculation of relative distances, the points for building the indoor navigation map and the relative distances between all points are obtained to form a relative distance matrix. Multidimensional scaling technology is used to calculate the relative spatial relationship between points as a non-similarity measurement parameter to build the indoor navigation map.

[0063] S3. Extract the indoor plane structure.

[0064] S31. Input a monocular image of an indoor space captured by a smartphone, and extract the three-dimensional layout structure of the monocular image through an indoor three-dimensional scene understanding network model.

[0065] The initial framework is trained through Convolutional Neural Networks (CNN) and used as feature vector to input into the structured support vector machine to improve the accuracy of indoor 3D spatial feature extraction. Among them, the indoor layout estimation based on CNN is to estimate the ordered set of indoor layout key points by directly training the room corner points (key points) and room types, and connect them in a specific order to obtain the indoor layout framework.

[0066] S32: Map the indoor 3D scene into a 2D image and perform projection transformation.

[0067] Since the length, width and height structural information of the three-dimensional space layout extracted from the mobile phone image is represented by pixel distance, the image will be distorted to varying degrees in the process of mapping the indoor 3D scene to the 2D image, resulting in distortion based on the spatial structure ratio. Therefore, it is necessary to restore the true proportional relationship between the length, width and height of the indoor space by eliminating the influence of image distortion.

[0068] The VPBC technology is used to extract the camera parameters based on the vanishing point. The camera principal point is located at the center of the image, and the camera intrinsic parameter matrix K is in the form of:

[0069]

[0070] f is the focal length, which can be calculated from the vanishing point: Let the vanishing point be vp0(x i0 ,y i0 ), vp1(x i1 ,y i1 ), vp2(x i2 ,y i2 ), since the centroid of the triangle formed by the three vanishing points is the intersection point p(ox,oy) of the camera optical axis and the imaging plane, the coordinates of vp0 in the camera coordinate system are (x c0 ,y c0 ,f), where x c0 =x i0 -ox,y c0 =-(y i0 -oy), the coordinates of the other two vanishing points can be obtained in the same way. Using the orthogonal relationship between the vanishing points, the focal length and rotation matrix can be obtained as follows:

[0071]

[0072] Calculate the projection transformation matrix H using camera parameters ij , you can use the vanishing point vp i and vp j The determined plane is corrected by “orthographic projection”, H ij The solution is as follows:

[0073] H ij =K·R ij ·K -1 ;

[0074] Using the transformation matrix H of the front wall and floor 01 and H 02 Reproject the 4 intersection points to get the coordinates of each point in the front view after eliminating distortion

[0075]

[0076] S33. Restore the spatial structure ratio and obtain the indoor plane structure.

[0077] Based on the re-projection The relative lengths of length, width and height are calculated based on the coordinates, and finally the true proportional relationship of length, width and height of the indoor space is solved:

[0078]

[0079] Assuming the distance between indoor rooms is the mean value H, the length, width and height extracted from the image are all calculated based on the building height H to ensure the scale consistency of the map.

[0080] S4. Integrate and optimize the navigation path and indoor plane structure.

[0081] S41, Corridor Construction: First, the following rules need to be followed during image acquisition: During the turning process, take pictures of all the view areas facing the turning points in a clockwise order. Figure 4 As shown: When the user passes a corner, he first takes a photo in the corridor in one direction, and then rotates to take a photo in the other direction.

[0082] The gyroscope and accelerometer record both angle and acceleration. The fluctuation of the gyroscope reading (up or down) indicates that the user turns left or right at this position. During the two fluctuations, the user takes pictures of the corridor, and the inertial data can be used to indicate the user's corner position. According to the image collection rules and the analysis of the shooting behavior, the corrected trajectory route is matched with the corridor plane structure extracted from the mobile phone image to realize the construction of the corridor area map.

[0083] S42, room construction: When collecting room images, the rules are as follows Figure 5 As shown in the figure, the user shoots from both sides of the room. Then, the room structure is drawn in the indoor map by matching the room plan structure extracted from the image with the inertial navigation data.

[0084] After the trajectory data is matched with the captured image position, the map accuracy is further optimized by constructing an energy function. The energy equation is proposed to be constructed as follows:

[0085] min x ={x f ,R,t}E l (x)+E c (x)+E b (x);

[0086] Where E l (x) represents the complexity of the layout, E c (x) represents the degree of closure, E b (x) represents the boundary similarity between adjacent segments, {x f ,R,t} represents the rotation and displacement of all corridor paths and room layouts when they are connected.

[0087] The uniqueness of corridor and room layouts is limited by the following formula.

[0088]

[0089] Therefore, the present invention adopts the above-mentioned method for automatically constructing indoor maps by integrating multi-source trajectories and mobile phone images, and proposes a high-efficiency, low-cost, and independent indoor map element plane structure extraction method, which uses pictures taken by mobile phones as data sources, does not need to reconstruct the geometric structure of indoor three-dimensional models, and does not require professional measurement equipment. Only monocular pictures of indoor scenes are taken by smart phones, and the plane structure of indoor map elements is obtained through visual scene understanding technology, which effectively improves the authenticity and accuracy of indoor map expression. Only different types of data collected by smart phones are integrated with multi-source trajectories and standardized mobile phone image acquisition, and a new geometric feature extraction method is used to achieve high-precision construction of indoor maps. The advantages of indoor space features of the whole and the part, the framework and the details are fully utilized to complement each other, and users do not need to walk through all locations in the indoor environment, which effectively saves the labor cost of indoor map drawing and reduces time and calculation.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for automatically constructing indoor maps by integrating multi-source trajectories and mobile phone images, characterized in that: The following steps are involved: S1, using smartphones to collect corridor, indoor images and multi-sensor data; S2, build indoor navigation map; S3, extracting indoor plane structure; S4. Integrate and optimize the navigation path and indoor plane structure.

2. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 1, characterized in that: S2 includes the following steps: S21, obtaining multi-source trajectory data by using pedestrian dead reckoning to collect multi-sensor data from the smartphone platform; S22, obtaining behavior landmarks through a behavior recognition algorithm, and clustering the behavior landmarks through a clustering algorithm based on Wi-Fi fingerprints; S23. Calculate the relative distances between the behavior landmarks to obtain an indoor navigation map.

3. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 2, characterized in that: S3 includes the following steps: S31, inputting a monocular image of an indoor space captured by a smartphone, and extracting a three-dimensional layout structure of the monocular image through an indoor three-dimensional scene understanding network model; S32, mapping the indoor 3D scene into the 2D image and performing projection transformation; S33. Restore the spatial structure ratio and obtain the indoor plane structure.

4. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 3, characterized in that: S32 includes the following steps: S321, using VPBC technology to extract and solve camera parameters based on vanishing point. The camera principal point is located at the center of the image. The camera intrinsic parameter matrix K is in the form of: f is the focal length, which can be found from the vanishing point; S322, set the vanishing point in the image coordinate system to be vp0(x i0 ,y i0 ), vp1(x i1 ,y i1 ), vp2(x i2 ,y i2 ), since the centroid of the triangle formed by the three vanishing points is the intersection point p(ox,oy) of the camera optical axis and the imaging plane, the coordinates of vp0 in the camera coordinate system are (x c0 ,y c0 ,f), where x c0 =x i0 -ox,y c0 =-(y i0 -oy), the coordinates of the other two vanishing points can be obtained in the same way. Using the orthogonal relationship between the vanishing points, the focal length and rotation matrix can be obtained respectively: S323, using camera parameters to calculate the projection transformation matrix H ij , you can use the vanishing point vp i and vp j The determined plane is corrected by "orthographic projection", H ij The solution is as follows: H ij =K·R ij ·K -1 ; Using the transformation matrix H of the front wall and floor 01 and H 02 Reproject the 4 intersection points to get the coordinates of each point in the front view after eliminating distortion 5. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 4, characterized in that: The actual proportions of the length, width, and height of the interior space are:

6. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 5, characterized in that: S4 includes the following steps: S41, corridor construction, matching the corrected trajectory route with the corridor plane structure extracted from the mobile phone image to realize map construction of the corridor area; S42, matching the room plan structure extracted from the image with the inertial navigation data to achieve drawing of the room structure in the indoor map.

7. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 6, characterized in that: By constructing an energy function to further optimize and ensure map accuracy, the energy equation is proposed to be constructed as follows: min x ={x f ,R,t}E l (x)+E c (x)+E b (x); Where E l (x) represents the complexity of the layout, E c (x) represents the degree of closure, E b (x) represents the boundary similarity between adjacent segments, {x f ,R,t} represents the rotation and displacement of all corridor paths and room layouts when they are connected.

8. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 7, characterized in that: The conditions that restrict the uniqueness of corridor and room layouts are:

9. The method for automatically constructing an indoor map by integrating multi-source trajectories and mobile phone images according to claim 8, characterized in that: The following rules are followed during image acquisition: when shooting indoor corridors, you need to take pictures of all the view areas facing the turning points in a clockwise order during the turning process; when shooting indoor rooms, you need to shoot from both sides of the room towards the opposite side.

Images