An indoor space auxiliary positioning method based on visual features

Abstract

The present invention discloses an indoor space auxiliary positioning method based on visual features. A single camera of a mobile phone is used to collect a video stream of an indoor space in real time, and each frame of the image is transmitted to a server in real time; the feature points of the image are extracted; key frames are screened out according to the feature points; the positions of the screened key frames and their feature points are tracked; sparse point cloud position information and odometer information are constructed, and then a dictionary database is formed; the current image to be tested is captured and matched with the key frames in the dictionary database in real time to obtain the position of the mobile phone at the time of the current image to be tested, and then the route trajectory of the mobile phone in the indoor space is obtained. The method of the present invention can effectively utilize visual features to calculate the movement trajectory of the mobile phone in real time under the condition of a mobile phone with only a built-in monocular camera, and generate a sparse point cloud map of the scene for auxiliary positioning, with good accuracy and robustness, effectively reducing the construction period and maintenance cost of the indoor positioning system.

Description

Technical Field

[0001] The present invention relates to an indoor image positioning method in the field of simultaneous positioning and mapping (SLAM), and in particular to an indoor space auxiliary positioning method based on visual features. Background Art

[0002] Smartphones have become an essential tool in modern life, bringing convenience to our lives. They are often equipped with high-definition cameras to meet people's daily photography needs. With the rapid adoption and widespread use of smartphones, mobile phone positioning technology has also attracted considerable attention. This is especially true indoors or in other complex environments with weak GPS signals, making it difficult to locate using base station signals. When a phone is indoors, satellite and base station signals may not be able to provide accurate positioning.

[0003] In recent years, China has explicitly stated the need to promote the development of perception and positioning technologies and enhance location-based services. Many researchers are already deploying perception technologies in fields such as robotics and autonomous driving. Utilizing visual features for assisted positioning can offset the inaccuracies of GPS and other positioning technologies in complex environments.

[0004] Therefore, in view of the defects of the existing technology, it is necessary to propose a technical solution to solve the technical problems of the existing technology by performing auxiliary positioning and generating sparse point clouds when framing and shooting on a mobile phone. Summary of the Invention

[0005] In order to solve the problems existing in the background technology, the present invention provides an indoor space auxiliary positioning method based on visual features, which solves the technical problems existing in the existing technology by framing and shooting on a mobile phone and generating a sparse point cloud, and is applied to the field of mobile phone perception and positioning.

[0006] This method requires no infrastructure, offers low complexity, and high universality, providing a new direction for future mobile phone positioning development. The method uses images captured by the smartphone's camera as it moves, calculating the device's steering angle and trajectory, while also generating a sparse point cloud map.

[0007] The technical solution of the present invention comprises the following steps:

[0008] Step S1: Use a single camera of a mobile phone to collect a video stream of an indoor space in real time, and transmit each frame of the video stream to a server in real time via a local area network during the video stream collection process; the camera is a monocular camera.

[0009] The initial frame is captured at a fixed, known location within the indoor space. A starting point is typically set, and the phone is moved from that point to capture the video stream.

[0010] Step S2: extracting feature points of each frame of the video stream from the server;

[0011] Step S3: Filter out key frames based on the feature points of the image;

[0012] Step S4: Tracking the positions of the selected key frames and their feature points to obtain the positions of the key frames and their feature points;

[0013] Step S5: adding the tracked key frames and their feature points to the map of the indoor space to form odometry and sparse point cloud location information. All key frames and their positions constitute the odometry information, and all key frame feature points and their positions constitute the sparse point cloud location information. The sparse point cloud location information and the odometry information are combined to generate the location information of the indoor space in the world coordinate system. A dictionary database is constructed from the sparse point cloud location information and the odometry information, and a dictionary database is constructed from all key frames and their feature points.

[0014] Step S6: Match the current image to be tested captured in real time with the key frames in the dictionary database to obtain the position of the mobile phone in the indoor space corresponding to the current image to be tested, and then obtain the route trajectory of the mobile phone in the indoor space during the shooting process to achieve auxiliary positioning in the indoor space.

[0015] The step S2 is specifically as follows:

[0016] Step S21: In a frame of image, a pixel whose brightness value and the difference between the brightness value and the pixel values of its surrounding neighborhood are greater than a brightness threshold is selected as a key point;

[0017] Step S22: storing the pixel value of each pixel in the neighborhood around each key point as the descriptor of the key point;

[0018] Step S23: A key point and its descriptor constitute a feature point, and the direction of the feature point is calculated using the grayscale centroid method.

[0019] The original feature points have no direction information. When the next frame image is rotated, the descriptor will also change, making the feature points not robust to rotation. The grayscale centroid method is used to calculate the direction of the feature points, which adds a direction to the feature points and makes them rotation invariant.

[0020] In step S3, the key frame is obtained by performing the following judgment based on the feature points obtained from the current frame image:

[0021] If the number of frames between the current frame image and the previous key frame is greater than the first judgment threshold, the distance between the current frame image and the previous key frame is greater than the second judgment threshold, and the number of feature points obtained in the current frame image is greater than the third judgment threshold, then the current frame image is used as the key frame;

[0022] Otherwise, the current frame image is not used as a key frame;

[0023] The distance between the current frame image and the previous key frame refers to the distance between the position of the mobile phone in the indoor space when the current frame image is captured and the position of the mobile phone in the indoor space when the previous key frame is captured.

[0024] The key frame refers to a representative frame among the local similar frames in the video stream.

[0025] The specific implementation uses the BA method to optimize the camera pose and key frames of the mobile phone, which is more accurate than the ICP or photometric error minimization method.

[0026] In step S4, the position of the mobile phone in the indoor space when the key frame is collected is used as the position of the key frame. A pure visual feature method is used on the server to calculate the change in the mobile phone camera posture between adjacent frames in real time based on the change relationship between the feature points of the current key frame and the feature points of the previous key frame. Then, the position of the current key frame is obtained by combining the change in the mobile phone camera posture with the position of the previous key frame.

[0027] In step S4, the position of the object target corresponding to the feature point in space is obtained by triangulation method based on the position relationship of the same feature point in each key frame image of adjacent key frames and the change of the mobile phone camera posture between adjacent frames as the position of the feature point of the key frame.

[0028] The step S5 is specifically as follows:

[0029] The feature points of the current key frame are stored in a vector container, and the pre-loaded dictionary database is called. The three-dimensional space similarity transformation method is used to calculate the similarity between the current key frame and the key frames in the dictionary database to establish a connection. Based on the similarity, the mobile phone camera posture change between the current image to be tested and the key frames in the dictionary database is calculated. Then, the position of the current image to be tested is obtained by combining the position of the key frames in the dictionary database, that is, the position of the mobile phone in the indoor space when the current image to be tested is obtained. The position of the current image to be tested, which is continuously captured, constitutes the route trajectory of the mobile phone shooting process.

[0030] The SerchBySim3 method is to match two separated key frames through a dictionary database, re-establish the connection, and make a soft transition to distribute the accumulated error to all key frames as much as possible.

[0031] The dictionary database is a bag-of-words dictionary structure and uses an improved binary storage method for compression processing. Loading a dictionary database in a conventional text file format is very slow. The improved binary storage format reduces the loading time from 8.37 seconds to 0.32 seconds.

[0032] In the specific implementation, the dictionary database of the bag-of-words dictionary structure trained offline is loaded in advance, and an improved binary storage method is used to compress the bag-of-words, which reduces storage space and improves loading speed.

[0033] The improved binary storage method is specifically as follows: traverse all offline images to be trained, extract the feature points of each frame of the image to be trained, store the image and its feature points in a binary form into a dictionary database, and finally save it as a bin file with faster loading speed and less program memory occupation.

[0034] Therefore, the present invention can achieve the effect of indoor space assisted positioning by relying solely on images captured by the monocular camera on the mobile phone, without depth images and accurate GPS.

[0035] The beneficial effects of the present invention are:

[0036] This paper proposes a method for assisting indoor positioning based on visual features that doesn't rely on satellite signals. Using a smartphone with a built-in monocular camera as a positioning device, this method effectively utilizes visual features to calculate the phone's trajectory in real time and generate a sparse point cloud map of the scene for auxiliary positioning.

[0037] This method can achieve positioning accuracy of 2 centimeters indoors or in other environments with weak GPS signals. By using image processing and the SLAM framework to calculate the mobile phone's trajectory and posture, the method offers excellent accuracy and robustness, effectively reducing the construction cycle and maintenance costs of indoor positioning systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flow chart of the method of the present invention;

[0039] Figure 2 It is a block diagram of the image acquisition and transmission scheme;

[0040] Figure 3 Filter the flow chart for keyframes;

[0041] Figure 4 It is a sparse point cloud location information map and the route trajectory of the mobile phone shooting process. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to the accompanying drawings and specific implementation examples. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are therefore only examples and are not intended to limit the scope of protection of the present invention.

[0043] The indoor space auxiliary positioning method of the present invention is not affected by satellite signals and base station signals, and can solve the problem of accurate positioning in indoor scenes to a certain extent.

[0044] See also Figure 1 , the embodiments of the present invention are as follows:

[0045] Step S1: Use a single camera on a mobile phone to capture a video stream, and transmit the captured image data to a server via a local area network;

[0046] See also Figure 2 It is a block diagram of the image acquisition and transmission scheme;

[0047] With the help of the camera configured on the smartphone, real-time video stream is collected. At the same time, the LAN interface of the smartphone is opened. The local server will actively connect to the smartphone through the LAN and receive the video stream data transmitted by the smartphone in real time through the LAN IP and port.

[0048] Step S2: extracting feature points of each frame image in the video stream from the server;

[0049] The number of feature points N that need to be extracted in the current frame is used to extract feature points of the current frame image using the image pyramid method. The number of image pyramid layers is m, the m value is 8, the image pyramid scaling factor is s, and the s value is That is, the current frame image is downsampled in a 1:1.2 scaling ratio to obtain a total of 8 images, where the width of the 0th layer image is W and the height is H, and the corresponding area is H*W=C. Then, the number of feature points of the obtained image pyramid layers is allocated respectively. The number of feature points that need to be extracted in the 0th layer is:

[0050]

[0051] The number of feature points in the i-th layer pyramid is:

[0052]

[0053] Furthermore, feature point extraction is performed on each layer of the image pyramid. Pixel p is selected in the current layer of the image pyramid, and its brightness is Ip. A threshold T is set. 16 pixels on a circle with a radius of 3 pixels are selected with pixel p as the center. If the brightness of 5 consecutive points is greater than Ip+T or less than Ip-T, the pixel p is used for the next descriptor calculation. Otherwise, the pixel is discarded and a new pixel is selected.

[0054] Select N point pairs around pixel p, and combine the comparison results of these N point pairs as descriptors. Then the pixel p with the descriptor is considered to be a feature point.

[0055] Furthermore, the grayscale centroid method is used to calculate the direction of the feature points. After adding the direction to the feature points, the feature points have rotation invariance.

[0056] Step S3: Filter out key frames in the video stream;

[0057] See also Figure 3 , the steps to screen suitable key frames are:

[0058] Step S31: determining whether the number of frames between the current frame image and the previous key frame is greater than a set first determination threshold value for the current frame extracted from the video stream;

[0059] Step S32: determining whether the distance between the current frame image and the previous key frame is greater than a second determination threshold;

[0060] Step S33: whether the number of feature points obtained in the current frame image is greater than a third determination threshold;

[0061] Step S34: If the above conditions are met, the current frame is determined to be a key frame, otherwise the current frame is discarded;

[0062] Step S4: tracking the positions of the selected key frames and their feature points;

[0063] According to the positional relationship of the same feature point in each key frame image in consecutive adjacent key frames and the change of the mobile phone camera posture between adjacent frames, the position of the object target corresponding to the feature point in space is obtained by triangulation method as the position of the feature point of the key frame, and the key frame and its feature point are tracked;

[0064] Step S5: adding the tracked key frames and their feature points to the map of the indoor space to form odometry and sparse point cloud location information. All key frames and their positions constitute the odometry information, and all key frame feature points and their positions constitute the sparse point cloud location information. A dictionary database is constructed from all key frames and their feature points.

[0065] Step S51: adding the tracked key frames and their positions to the map of the indoor space to form odometer information;

[0066] Step S52: adding the tracked feature points and their positions of the key frames to the map of the indoor space to form sparse point cloud position information;

[0067] Step S53: constructing a dictionary database from all key frames and their feature points;

[0068] Step S6: Match the current image to be tested captured in real time with the key frames in the dictionary database to obtain the position of the mobile phone in the indoor space corresponding to the current image to be tested, and then obtain the route trajectory of the mobile phone in the indoor space during the shooting process to achieve auxiliary positioning in the indoor space.

[0069] Step S61: calling a pre-loaded dictionary database and using the three-dimensional space similarity transformation method SerchBySim3 to calculate the similarity between the current key frame to be tested and the key frames in the dictionary database;

[0070] Step S62: calculating the change in the camera pose of the mobile phone between the current image to be tested and the key frames in the dictionary database based on the similarity;

[0071] Step S63: The position of the current image to be tested is obtained by using the change in the camera posture of the mobile phone in combination with the position of the key frame in the dictionary database, that is, the position of the mobile phone in the indoor space when the current image to be tested is obtained. The position of the current image to be tested that is continuously captured constitutes the route trajectory of the mobile phone shooting process.

[0072] See also Figure 4 , which are sparse point cloud location information maps and the route trajectory of the mobile phone shooting process. The left picture is the route trajectory of the mobile phone shooting process, and the right picture is the sparse point cloud location information map. The points in the figure represent map points, the boxes represent key frames, and the constraints between key frames represent the connection relationship between key frames;

[0073] This invention addresses the difficulty of accurately positioning indoors using base station signals and other methods. This visual feature-based indoor positioning method infers the phone's position and the trajectory it took during the capture process based on changes in feature points in key frames, demonstrating its universality and robustness. This method is applicable not only to indoor spaces but also to assisting positioning of mobile phones in other complex environments.

Claims

1. An indoor space auxiliary positioning method based on visual features, characterized in that: The steps include: Step S1: Use a single camera of a mobile phone to collect a video stream of the indoor space in real time, and transmit each frame of the video stream to the server in real time via the local area network during the video stream collection process; Step S2: extracting feature points of each frame of the video stream from the server; Step S3: Filter out key frames based on the feature points of the image; Step S4: tracking the positions of the selected key frames and their feature points; Step S5: adding the key frames and their feature points to the map of the indoor space to form odometry and sparse point cloud location information. All key frames and their positions constitute the odometry information, and all key frame feature points and their positions constitute the sparse point cloud location information. A dictionary database is constructed using the sparse point cloud location information and the odometry information. Step S6: Match the current image to be tested captured in real time with the key frames in the dictionary database to obtain the position of the mobile phone in the indoor space corresponding to the current image to be tested, and then obtain the route trajectory of the mobile phone in the indoor space during the shooting process, thereby realizing auxiliary positioning in the indoor space; In step S4, the position of the mobile phone in the indoor space when the key frame is captured is used as the position of the key frame. A pure visual feature method is used on the server to calculate the change in the mobile phone camera posture between adjacent frames in real time based on the change relationship between the feature points of the current key frame and the feature points of the previous key frame. The position of the current key frame is then obtained by combining the change in the mobile phone camera posture with the position of the previous key frame. In step S4, the position of the object target corresponding to the feature point in space is obtained by triangulation method based on the position relationship of the same feature point in each key frame image of adjacent key frames and the change of the mobile phone camera posture between adjacent frames as the position of the feature point of the key frame.

2. The indoor space auxiliary positioning method based on visual features according to claim 1, characterized in that: The step S2 is specifically as follows: Step S21: In a frame of image, a pixel whose brightness value and the difference between the brightness value and the pixel values of its surrounding neighborhood are greater than a brightness threshold is selected as a key point; Step S22: storing the pixel value of each pixel in the neighborhood around each key point as the descriptor of the key point; Step S23: A key point and its descriptor constitute a feature point, and the direction of the feature point is calculated using the grayscale centroid method.

3. The indoor space auxiliary positioning method based on visual features according to claim 1, characterized in that: In step S3, the key frame is obtained by performing the following judgment based on the feature points obtained from the current frame image: If the number of frames between the current frame image and the previous key frame is greater than the first judgment threshold, the distance between the current frame image and the previous key frame is greater than the second judgment threshold, and the number of feature points obtained in the current frame image is greater than the third judgment threshold, then the current frame image is used as the key frame; Otherwise, the current frame image is not used as a key frame; The distance between the current frame image and the previous key frame refers to the distance between the position of the mobile phone in the indoor space when the current frame image is captured and the position of the mobile phone in the indoor space when the previous key frame is captured.

4. The indoor space auxiliary positioning method based on visual features according to claim 1, characterized in that: The step S5 is specifically as follows: The feature points of the current key frame are stored in a vector container, and the pre-loaded dictionary database is called. The similarity between the current key frame and the key frames in the dictionary database is calculated using the three-dimensional space similarity transformation method. The mobile phone camera posture change between the current image to be tested and the key frames in the dictionary database is calculated based on the similarity. Then, the position of the current image to be tested is obtained by combining the position of the key frames in the dictionary database with the posture change of the mobile phone camera. The position of the current image to be tested collected by continuous shooting constitutes the route trajectory of the mobile phone shooting process.

5. The indoor space auxiliary positioning method based on visual features according to claim 1, characterized in that: The dictionary database is a bag-of-words dictionary structure and uses an improved binary storage method for compression processing.

Images