Zero-deployment indoor precise positioning method based on fusion of optical flow and inertial navigation

Abstract

The present invention relates to a zero-deployment indoor precise positioning method that integrates optical flow and inertial navigation. The present invention combines the advantages of inertial sensors and cameras, and utilizes the direction calculated by the inertial sensor and the displacement calculated by the camera to form a zero-deployment indoor precise positioning method that integrates optical flow and inertial navigation. This indoor positioning method can accurately calculate the walking direction and walking displacement, thereby accurately calculating the current position. The present invention is based on experiments conducted on mobile phone inertial sensor and camera data, but the present invention can be applied to other terminal devices that include inertial sensors and cameras, such as tablet computers, robotic cars, etc.

Description

Technical Field

[0001] The present invention relates to an indoor positioning method that does not rely on additional infrastructure, and in particular to a zero-deployment indoor precise positioning method that integrates optical flow and inertial navigation, and belongs to the field of the Internet of Things. Background Art

[0002] Most social behaviors are closely related to indoor locations. Accurate indoor positioning is a crucial cornerstone of the digitalization of society and will revolutionize how we live and work. Indoor positioning services, as a fundamental service, continuously provide precise location information. Applications include, but are not limited to, security monitoring, emergency rescue, intelligent care, precision marketing, and smart factories. While the Beidou satellite navigation system and the Galileo positioning system can accurately locate mobile devices in open, lightly obstructed outdoor environments, these traditional positioning systems fall short of meeting human needs in indoor environments and densely populated areas due to multipath and shadowing effects caused by buildings.

[0003] Common indoor positioning technologies can be divided into two categories based on whether or not infrastructure is deployed. Infrastructure-based positioning methods, such as those based on radio signals like WiFi, rely on pre-deployed wireless access points and may fail in extreme situations such as fires and earthquakes. Furthermore, the complex and variable multipath effects caused by the dynamic changes in the indoor environment lead to large positioning errors. A significant disadvantage shared by these infrastructure-based methods is that the cost of equipment deployment increases proportionally with the indoor area, resulting in higher costs for positioning methods covering a wide range. The high or low deployment cost will affect its popularity and speed, even if its positioning effect is good. Common zero-infrastructure positioning methods rely on inertial sensors carried by positioning devices. However, the inertial sensors used in non-dedicated positioning devices (such as smartphones) are relatively low-cost and have low accuracy. More importantly, accumulated errors can easily lead to a continuous increase in positioning errors.

[0004] The coordinate system used by the accelerometer, gyroscope, and magnetometer is the top-right coordinate system. For the top-right coordinate system, when the device is lying face-up on a table, the coordinate system is defined relative to the device screen, with the x-axis pointing to the right of the device screen, the y-axis pointing to the front of the device screen, and the z-axis extending perpendicular to the device screen and upward. The coordinate axes do not translate when the device's screen orientation changes; that is, the sensor's coordinate system does not change as the device moves. The accelerometer measures acceleration applied to the device, including gravity. The gyroscope measures rotational rate about the device's x, y, and z axes, with a positive direction being counterclockwise. If an observer looks at the device at the origin from a positive position on the x, y, or z axis, the observer will report a positive rotation if the device appears to be rotating counterclockwise. The magnetometer measures magnetic field strength along three axes and can monitor changes in the Earth's magnetic field.

[0005] Corner points can be simply regarded as image pixels with significant features. Corner point detection, also known as feature point detection, is a method used in image processing, computer vision and other fields to obtain information such as local feature points of an image. It has broad application prospects in motion detection, image matching, video tracking and other fields.

[0006] Optical flow was first proposed by American psychologist Gibson in the 1940s. Optical flow is the apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between the observer (such as the eyes or camera) and the scene. This relative motion can include observer motion, scene motion, or both. Optical flow estimation is an important area of ​​computer vision research, with a wide range of applications, including object segmentation, motion detection, and stereo disparity measurement. Smartphone optical flow estimation uses the spatial and temporal correlation and changes between two consecutive frames of a continuous image sequence captured by an image sensor to calculate the velocity vector of a pixel in the two adjacent frames, known as optical flow. Calculating optical flow for every pixel in an image is called dense optical flow, while calculating optical flow for a subset of pixels (i.e., corner points with distinct features) is called sparse optical flow. Summary of the Invention

[0007] The purpose of the present invention is to make full use of various sensors of mobile devices for position positioning, thereby achieving indoor positioning without relying on additional infrastructure and achieving higher positioning accuracy.

[0008] In order to achieve the above objectives, the technical solution of the present invention is to provide a zero-deployment indoor precise positioning method that integrates optical flow and inertial navigation, which is characterized by comprising the following steps:

[0009] Step 1. Get the initial position x0, y0 by scanning the QR code;

[0010] Step 2: Obtain a pixel distance ratio r by scanning the QR code, wherein the QR code has its actual width encoded in advance, and the pixel distance ratio is the actual width of the QR code divided by the pixel width of the QR code;

[0011] Step 3: Calculate the initial attitude. When the terminal is stationary, the initial attitude of the terminal device is calculated by reading the accelerometer, gyroscope, and magnetometer, including the pitch angle, roll angle, and yaw angle.

[0012] Step 4: Get the calculated attitude change through the gyroscope reading;

[0013] Step 5: Get the estimated attitude change through the accelerometer reading and the magnetometer reading;

[0014] Step 6: Update the posture by fusing the calculated posture change and the estimated posture change to obtain the current posture;

[0015] Step 7: Perform attitude compensation on the image of the ground captured by the camera. The attitude compensation requires the pitch angle and roll angle of the device attitude.

[0016] Step 8: Divide the image into upper and lower regions based on the forward speed of the device. The pixel height of the upper region is calculated as follows:

[0017]

[0018] Where h is the pixel height of the upper region, h0 is the pixel height of the video frame, r is the pixel distance ratio, ν is the y-axis optical flow calculated in the previous frame, and Δt is the video frame interval.

[0019] Step 9: Detect the corner points of the upper and lower area images respectively, and use the Lucas sparse optical flow algorithm to calculate the positions of these corner points in the next frame to obtain the optical flow;

[0020] Step 10: Eliminate part of the optical flow obtained in step 9 based on the motion features;

[0021] Step 11: Apply median filtering to the optical flows of the upper and lower regions respectively, and then apply mean filtering to the optical flows of the upper and lower regions to obtain the final optical flow;

[0022] Step 12: Combine the pixel distance ratio r to convert the final optical flow into the displacement of the device in the real world;

[0023] Step 13: Update the device position at the previous moment to the current moment using the yaw angle and displacement. The calculation formula is as follows:

[0024]

[0025] Where μ i ,ν i represents the optical flow estimation of the i-th frame of the video, ψ i represents the yaw angle of the mobile device at the i-th frame of the video, r represents the pixel distance ratio, x0, y0 represents the initial position obtained through the QR code, x k ,y k Indicates the current location;

[0026] Step 14: Repeat steps 4 to 13 during the positioning process.

[0027] Preferably, in step 2, the pixel distance ratio is obtained by the following method:

[0028] Use a camera to shoot QR code video frames of a certain length of time, calculate the pixel distance ratio of each QR code video frame, and the final pixel distance ratio is the average value of the pixel distance ratios of the QR code video frames of a certain length of time.

[0029] Preferably, in step 5, after filtering the accelerometer readings by applying a low-pass filter, the estimated attitude change is obtained based on the filtered accelerometer readings.

[0030] Preferably, the cutoff frequency of the low-pass filter is 2 Hz, and the calculation formula is as follows:

[0031]

[0032] Where a′ t represents the filtered acceleration at the current moment, f represents the cutoff frequency, Δt represents the sampling period, a′ t-1 Indicates the filtered acceleration at the previous moment, a t Represents the raw acceleration.

[0033] Compared with the prior art, the present invention is beneficial in that:

[0034] This invention combines the advantages of inertial sensors and cameras. Using the direction calculated by the inertial sensors and the displacement calculated by the camera, it develops a zero-deployment indoor precision positioning method that integrates optical flow and inertial navigation. This indoor positioning method can accurately calculate walking direction and displacement, thereby accurately calculating the current position. This invention is based on experiments using data from mobile phone inertial sensors and cameras, but it can also be applied to other terminal devices that include inertial sensors and cameras, such as tablets and robotic vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a system structure diagram of the present invention;

[0036] Figure 2 A schematic diagram of the process of using the present invention;

[0037] Figure 3 A schematic diagram of scanning a QR code according to the present invention;

[0038] Figure 4 A trajectory diagram of a person walking in a straight line according to the present invention;

[0039] Figure 5 A rectangular walking trajectory diagram of the present invention;

[0040] Figure 6 A walking track diagram according to arrows of the present invention;

[0041] Figure 7 A trajectory diagram of a circular walk according to the present invention;

[0042] Figure 8 This is a cumulative distribution diagram of positioning errors when walking in a straight line according to the present invention;

[0043] Figure 9 This is a cumulative distribution diagram of positioning errors according to the rectangular walking method of the present invention;

[0044] Figure 10 This is a cumulative distribution diagram of positioning errors when walking according to the arrows of the present invention;

[0045] Figure 11 This is a cumulative distribution diagram of positioning errors of the present invention according to circular walking. DETAILED DESCRIPTION

[0046] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0047] This embodiment discloses a zero-deployment indoor precise positioning method that integrates optical flow and inertial navigation, specifically comprising the following steps:

[0048] Step 1: Scan the QR code to obtain the initial position. The QR code pre-encodes its location. Use OpenCV 4.4.0 to recognize the QR code image captured by the camera and identify the initial position information, such as (0,0).

[0049] Step 2: Scan the QR code to obtain the pixel distance ratio. The QR code pre-encodes its actual width. Use a camera to capture the QR code on the ground for 2 seconds. Use OpenCV 4.4.0 to identify the pixel positions of the QR code vertices in the image captured by the camera. The pixel distance ratio for each frame is calculated by dividing the actual width of the QR code by the pixel width of the QR code. The final pixel distance ratio is the average pixel distance ratio of the 2 seconds of video frames.

[0050] Step 3: Calculate the initial attitude. When the terminal is stationary, the initial attitude of the terminal device is calculated by reading the accelerometer, gyroscope and magnetometer, including the pitch angle, roll angle and yaw angle. The calculation formulas are shown in equations (1) and (2):

[0051]

[0052]

[0053] In formula (1), θ represents the pitch angle, φ represents the roll angle, g represents the acceleration due to gravity, and a x ,a y ,a z Indicates the reading of the accelerometer. In formula (2), ψ represents the yaw angle, h x ,h y ,h zRepresents the components of the three axes of the Earth's magnetic field, m x ,m y ,m z Represents the magnetometer reading.

[0054] Step 4: Calculate the attitude change through the gyroscope readings and use the Runge-Kutta method to convert the three-axis gyroscope readings into quaternion changes.

[0055] Step 5: Apply a low-pass filter to the accelerometer reading to remove some of the motion acceleration and obtain the acceleration more accurately. The cutoff frequency used by the low-pass filter is 2 Hz. The calculation formula is shown in formula (3):

[0056]

[0057] In formula (3), a′ t represents the filtered acceleration at the current moment, f represents the cutoff frequency, Δt represents the sampling period, a′ t-1 Indicates the filtered acceleration at the previous moment, a t Represents the raw acceleration.

[0058] Step 6: Estimate the attitude change using the filtered accelerometer and magnetometer readings, where the gradient is calculated using the Jacobian matrix and the step size is adaptively adjusted based on the original acceleration amplitude. In this embodiment, the formula for adaptive step size adjustment is shown in Equation (4):

[0059]

[0060] In formula (4), μ t represents the step size of gradient descent, a t represents the original acceleration, and g represents the acceleration due to gravity.

[0061] Step 7: Update the posture by fusing the calculated posture change and the estimated posture change to obtain the current posture.

[0062] Step 8: Perform attitude compensation on the image of the ground captured by the camera. The attitude compensation requires the pitch angle and roll angle of the device attitude.

[0063] Step 9: Divide the image into upper and lower regions based on the forward speed of the device. When the forward speed increases, the pixel height of the upper region decreases accordingly. The calculation formula for the pixel height of the upper region is shown in formula (5):

[0064]

[0065] In formula (5), h is the pixel height of the upper region, h0 is the pixel height of the video frame, r is the pixel distance ratio, ν is the y-axis optical flow calculated in the previous frame, and Δt is the video frame interval.

[0066] Step 10: Use the Shi-Tomasi corner detection algorithm to detect the corners of the upper and lower area images respectively, and use the Lucas sparse optical flow algorithm to calculate the positions of these corners in the next frame to obtain the optical flow.

[0067] Step 11: Eliminate some of the optical flows obtained in step 10 based on motion features, such as backward optical flow, optical flow with too large an angle, and optical flow with too large a speed. In this embodiment, optical flows with angles exceeding 20° will be eliminated, and optical flows with speeds exceeding 1.8 m / s will be eliminated.

[0068] Step 12: Apply median filtering to the optical flows of the upper and lower regions respectively, and then apply mean filtering to the optical flows of the upper and lower regions to obtain the final optical flow.

[0069] Step 13: Combine the pixel distance ratio to convert the final optical flow into the displacement of the device in the real world.

[0070] Step 14: Update the device position at the previous moment to the current moment position through the yaw angle and displacement. The calculation formula is shown in formula (6):

[0071]

[0072] In formula (6), μ i ,ν i represents the optical flow estimation of the i-th frame of the video, ψ i represents the yaw angle of the mobile device at the i-th frame of the video, r represents the pixel distance ratio, x0, y0 represents the initial position obtained through the QR code, x k ,y k Indicates the current location.

[0073] Step 15: Repeat steps 4 to 14 during the positioning process.

[0074] This invention is based on experiments conducted using data from mobile phone inertial sensors and cameras, but it can be applied to other terminal devices that include inertial sensors and cameras, such as tablet computers, robotic cars, etc.

Claims

1. A zero-deployment indoor precise positioning method integrating optical flow and inertial navigation, characterized in that: The following steps are involved: Step 1. Get the initial position x0, y0 by scanning the QR code; Step 2: Obtain a pixel distance ratio r by scanning the QR code, wherein the QR code has its actual width encoded in advance, and the pixel distance ratio is the actual width of the QR code divided by the pixel width of the QR code; Step 3: Calculate the initial attitude. When the terminal is stationary, the initial attitude of the terminal device is calculated by reading the accelerometer, gyroscope, and magnetometer, including the pitch angle, roll angle, and yaw angle. Step 4: Get the calculated attitude change through the gyroscope reading; Step 5: Get the estimated attitude change through the accelerometer reading and the magnetometer reading; Step 6: Update the posture by fusing the calculated posture change and the estimated posture change to obtain the current posture; Step 7: Perform attitude compensation on the image of the ground captured by the camera. The attitude compensation requires the pitch angle and roll angle of the device attitude. Step 8: Divide the image into upper and lower regions based on the forward speed of the device. The pixel height of the upper region is calculated as follows: Where h is the pixel height of the upper region, h0 is the pixel height of the video frame, r is the pixel distance ratio, ν is the y-axis optical flow calculated from the previous frame, and Δt is the video frame interval; Step 9: Detect the corner points of the upper and lower area images respectively, and use the Lucas sparse optical flow algorithm to calculate the positions of these corner points in the next frame to obtain the optical flow; Step 10: Eliminate part of the optical flow obtained in step 9 based on the motion features; Step 11: Apply median filtering to the optical flows of the upper and lower regions respectively, and then apply mean filtering to the optical flows of the upper and lower regions to obtain the final optical flow; Step 12: Combine the pixel distance ratio r to convert the final optical flow into the displacement of the device in the real world; Step 13: Update the device position at the previous moment to the current moment using the yaw angle and displacement. The calculation formula is as follows: Where μ i ,ν i represents the optical flow estimation of the i-th frame of the video, ψ i represents the yaw angle of the mobile device at the i-th frame of the video, r represents the pixel distance ratio, x0, y0 represents the initial position obtained through the QR code, x k ,y k Indicates the current location; Step 14: Repeat steps 4 to 13 during the positioning process.

2. The zero-deployment indoor precise positioning method integrating optical flow and inertial navigation as claimed in claim 1, characterized in that: In step 2, the pixel distance ratio is obtained using the following method: Use a camera to shoot QR code video frames of a certain length of time, calculate the pixel distance ratio of each QR code video frame, and the final pixel distance ratio is the average value of the pixel distance ratios of the QR code video frames of a certain length of time.

3. The zero-deployment indoor precise positioning method integrating optical flow and inertial navigation as claimed in claim 1, characterized in that: In step 5, after filtering the accelerometer readings using a low-pass filter, the estimated attitude change is obtained based on the filtered accelerometer readings.

4. The zero-deployment indoor precise positioning method integrating optical flow and inertial navigation as claimed in claim 3, characterized in that: The cutoff frequency of the low-pass filter is 2 Hz, and the calculation formula is as follows: Where a' t represents the filtered acceleration at the current moment, f represents the cutoff frequency, Δt represents the sampling period, a' t-1 Indicates the filtered acceleration at the previous moment, a t Represents the raw acceleration.

Images