Optical flow-camera calibration fusion visible light positioning method

Abstract

The present invention discloses an optical flow-camera calibration fusion visible light positioning method, which includes building a visible light positioning system indoors and using a camera to obtain image information containing white light LEDs at different times. n , for image information l n Preprocessing is performed to obtain the two-dimensional coordinates of the white light LED in the pixel plane, and the camera calibration and positioning algorithm is used to calculate the camera's pose in the world coordinate system. It is determined whether the camera calibration and positioning algorithm can obtain an analytical solution. If so, the optical flow algorithm is used to calculate the camera's pose, and the poses calculated by the camera calibration and positioning algorithm and the optical flow algorithm are accumulated. Then, the Kalman filter is used to perform data fusion to update the camera's pose information and complete the positioning of the visible light. If not, the optical flow algorithm is used to estimate the camera's pose, and the Kalman filter fusion is performed in combination with the camera's current position, moving speed and direction to update the camera's pose information and complete the positioning of the visible light.

Description

Technical Field

[0001] The present invention belongs to the technical field of indoor positioning and relates to a visible light positioning method integrating optical flow and camera calibration. Background Art

[0002] With the rapid development of visible light communication (VLC), visible light communication (VLC) technology is considered a highly promising indoor positioning solution. VLC uses light in the visible light band as an information carrier to transmit optical signals directly through the air. VLC can directly utilize existing lighting infrastructure, providing positioning services while also providing lighting, significantly reducing positioning costs. VLC has a wider spectrum resource, greatly alleviating the current shortage of wireless communication spectrum resources. VLC systems do not generate any radio frequency or electromagnetic interference and can be deployed in special environments with strict radio frequency radiation restrictions. VLC cannot penetrate opaque materials such as walls, thus providing high security.

[0003] The unique feature of indoor visible light positioning is that it uses LEDs as signal light sources, providing both illumination and data transmission. Using the indoor space as the transmission channel, the required positioning information is modulated using appropriate modulation techniques at the transmitting end of the LED light source. This information is then encoded into the electrical signal and loaded onto the LED. By flashing the LED at high speeds, which are invisible to the naked eye, the LED transmits the information-carrying optical signal into the channel, ultimately enabling positioning.

[0004] Visible light positioning (VLP) technology offers promising application prospects in indoor positioning due to its low cost, high accuracy, lack of electromagnetic interference, easy deployment, and the ability to balance communication positioning and illumination. With the continuous development of VLP technology, high-precision and high-performance positioning methods have emerged one after another. For example, Jia Songlin et al. proposed a "high-coverage camera-assisted indoor visible light positioning algorithm." This method, based on a nonlinear camera-assisted received signal strength algorithm, achieves high-precision indoor visible light positioning. However, this method cannot achieve accurate positioning when there is insufficient LED light. Bai Bo et al. proposed a "LED-based camera calibration wireless positioning algorithm," which uses a camera calibration positioning algorithm for indoor positioning, achieving centimeter-level accuracy and 5°-level attitude angle information. However, this algorithm cannot obtain an analytical solution and cannot accurately locate objects in low-light conditions. Prasan A et al. proposed an indoor visible light positioning algorithm using three LEDs, achieving high positioning accuracy at the millimeter level in computer simulations. However, this algorithm suffers from poor real-time performance and is difficult to generalize in everyday scenarios. Summary of the Invention

[0005] The purpose of the present invention is to provide an optical flow-camera calibration fusion visible light positioning method, which solves the problem that existing visible light positioning methods cannot accurately position when there are few light sources.

[0006] The technical solution adopted by the present invention is a method for integrating optical flow and camera calibration with visible light positioning, comprising the following steps:

[0007] Step 1: Build a visible light positioning system indoors;

[0008] Step 2: Use the camera to obtain image information containing white light LED at different times. n ;

[0009] Step 3: image information l n Perform preprocessing to obtain the two-dimensional coordinates of the white light LED in the pixel plane;

[0010] Step 4: Calculate the camera's position in the world coordinate system using a camera calibration and positioning algorithm based on the geometric relationship between the three-dimensional coordinates of the white light LED in the world coordinate system and the two-dimensional coordinates in the pixel plane.

[0011] Step 5: Determine whether the camera calibration and positioning algorithm can obtain an analytical solution. If so, use the optical flow algorithm to calculate the camera's pose, accumulate the poses calculated by the camera calibration and positioning algorithm and the optical flow algorithm, and then use the Kalman filter to perform data fusion to update the camera's pose information and complete the positioning of the visible light. If not, use the optical flow algorithm to estimate the camera's pose, combine the camera's current position, moving speed and direction, perform Kalman filter fusion, update the camera's pose information, and complete the positioning of the visible light.

[0012] The technical feature of the present invention is that, in step 3, the image information l n Preprocessing is performed, including threshold segmentation, image denoising and centroid positioning, to obtain a binary image containing the white light LED. The two-dimensional coordinates of the white light LED in the pixel plane are calculated through the binary image.

[0013] In step 5, the poses calculated by the camera calibration and positioning algorithm and the optical flow algorithm are accumulated, including taking the pose calculated by the optical flow algorithm as prediction information, accumulating the pose calculated by the camera calibration and positioning algorithm as update information, and accumulating the accumulated three-dimensional coordinates of the camera in the world coordinate system as the translation matrix and the accumulated three attitude angles as the rotation matrix.

[0014] In step 1, a visible light positioning system is built indoors, including installing white light LED lights, cameras, and computers that receive and process camera image information indoors.

[0015] In step 2, the camera is used to obtain image information containing white light LED at different times. n Before the white light LEDs are detected, the frequency of each white light LED is modulated to achieve accurate identification of different white light LEDs.

[0016] The beneficial effects of the present invention are as follows: the user's preliminary positioning is achieved by adopting the camera calibration and positioning algorithm and the optical flow algorithm through the information of the light source; the positioning results of the optical flow algorithm and the camera calibration and positioning algorithm are fused by Kalman filtering, so that the accuracy of the positioning result is greatly improved, and the positioning accuracy can reach the millimeter level; when there are few light sources and the camera calibration and positioning algorithm cannot obtain an analytical solution, the camera positioning is achieved by the optical flow algorithm, and the accurate positioning of the camera is achieved under the condition of few light sources; the camera is used to collect information of indoor white light LEDs, which is easy to use, not subject to electromagnetic interference, not easily affected by the environment, easy to install, strong in durability, small in size, and low in cost; the visible light positioning method of the present invention is suitable for various complex indoor places and has wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of the optical flow-camera calibration fusion visible light positioning method of the present invention;

[0018] Figure 2 Schematic diagram of the process of the camera calibration and positioning algorithm in the optical flow-camera calibration fusion visible light positioning method of the present invention;

[0019] Figure 3 Schematic diagram of positioning results using the method of the present invention and using the camera calibration positioning algorithm alone when the white light LED is blocked;

[0020] Figure 4 This is a graph showing the relationship between positioning error and time when the white light LED is blocked and calculated using the method of the present invention and using the optical flow algorithm alone;

[0021] Figure 5 This is a CDF diagram of the positioning error using the method of the present invention when the white light LED is blocked;

[0022] Figure 6 Schematic diagram of positioning results when white light LED is unobstructed and at a large angle using the method of the present invention and using the camera calibration positioning algorithm and the optical flow algorithm alone;

[0023] Figure 7 This is a graph showing the relationship between positioning error and time when the white light LED is unobstructed and the positioning method of the present invention is used, and when the camera calibration positioning algorithm and the optical flow algorithm are used alone.

[0024] Figure 8 This is a CDF diagram of the positioning error of the method of the present invention when the white light LED is not blocked and at a large rotation angle. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] The present invention provides a visible light positioning method integrating optical flow and camera calibration, comprising the following steps:

[0027] Step 1: Build a visible light positioning system indoors. This involves installing four white LEDs on top of a 3m×3m×2m test platform. The positions are (30cm, 130cm, 190cm), (30cm, 30cm, 190cm), (130cm, 30cm, 190cm), and (130cm, 130cm, 190cm). Place an OpenMV camera and a computer in the middle of the test platform. The parameters of the visible light positioning system are shown in Table 1:

[0028] Table 1

[0029]

[0030] The positioning method based on the image sensor uses the geometric relationship between the image sensor and the LED to calculate the position coordinates of the camera.

[0031] Step 2: Initialize the basic parameters of the camera. To accurately identify different LED lights, modulate the frequencies of the four LEDs at 3KHz, 4KHz, 5KHz, and 6KHz respectively. Send the modulated ID through the white light LEDs and use the camera to obtain image information containing the white light LEDs at different times. n , the camera images the captured white light LED as alternating light and dark stripes containing LED-ID information.

[0032] In step 3, the computer processes the received LED image containing light and dark stripes to obtain the LED ID information. That is, the LED is first identified and aligned through the LED light automatic detection and tracking algorithm, and then the LED-ID recognition algorithm is used to identify different ID position information.

[0033] Image information l n Perform preprocessing, including threshold segmentation, image denoising, and centroid positioning on the image, to obtain a binary image containing the white light LED, and calculate the two-dimensional coordinates of the white light LED in the pixel plane through the binary image;

[0034] Step 4: Calculate the camera's position in the world coordinate system using a camera calibration and positioning algorithm based on the geometric relationship between the three-dimensional coordinates of the white light LED in the world coordinate system and the two-dimensional coordinates in the pixel plane.

[0035] The camera calibration and positioning algorithm is a high-precision positioning algorithm based on visible light signals and cameras. This algorithm uses existing lighting equipment as the transmitter and the camera as the receiver. By obtaining the two-dimensional coordinates of the white light LED in the pixel plane, constraint equations are established for the two-dimensional coordinates and the coordinates of the white light LED in the world coordinate system, and finally the coordinates of the camera in the world coordinate system are determined. At the same time, the camera's posture information can also be generated.

[0036] The main steps of the camera calibration and positioning algorithm are as follows: first, initialize the basic parameters of the camera, obtain the position coordinates of the center of the white light LED in the pixel coordinate system, and use the Newton-Raphson method to approximate the camera position. Finally, obtain the position and attitude information of the camera in the world coordinate system. For the specific process, see Figure 2 .

[0037] The image sensor obtains the coordinates [U, V] of the white light LED in the pixel coordinate system:

[0038] [U,V] T =h1(X”,Y”)=[f x ·X”+C X ,f y ·Y”+C Y ] T (1)

[0039]

[0040] [X',Y',Z'] T =h3( c X, c Y, c Z)=[ c X / c Z, c Y / c Z,(X') 2 +(Y') 2 ] T (3)

[0041]

[0042] Where: c P is the position of the white light LED in the camera coordinate system, [ N X, N Y, N Z] is the position of the white light LED in the world coordinate system, [ N X c , N Y c , N Z c ] is the position of the camera in the world coordinate system, f x,f y are the equivalent focal lengths of the x-axis and y-axis respectively, p1, p2 and k1, k2 are the radial and tangential distortions of the x-axis and y-axis respectively, and the rest are intermediate variables.

[0043] The camera calibration and positioning algorithm converts the world coordinate system into the camera coordinate system using the transformation matrix As shown in formula (5):

[0044]

[0045] In the formula: Symbol θ and ψ represent the camera's yaw, pitch, and roll angles, respectively. The camera calibration and positioning algorithm can accurately determine the camera's position in the world coordinate system with centimeter-level positioning accuracy. It can also generate the camera's attitude information. However, the camera calibration and positioning algorithm cannot obtain an analytical solution when the number of LEDs is insufficient.

[0046] Step 5: Determine whether the camera calibration and positioning algorithm can find an analytical solution. If so, the optical flow algorithm is used to estimate the camera's pose and use this as prediction information. The pose calculated by the camera calibration and positioning algorithm is accumulated as update information, where the accumulated three-dimensional coordinates of the camera in the world coordinate system are set as the translation matrix, and the accumulated three attitude angles are set as the rotation matrix. The Kalman filter is then used to fuse the translation and rotation matrices of the two positioning algorithms to update the camera's pose information. If not, the optical flow algorithm is used alone to calculate and accumulate the camera's pose. The Kalman filter is then used to fuse the Kalman filter information based on the camera's current position, movement speed, and direction to update the camera's pose information and complete visible light positioning.

[0047] The optical flow algorithm uses a non-iterative method to solve optical flow using the least squares method. The algorithm is based on the three assumptions of constant brightness, temporal continuity, and spatial consistency. In the first frame, the pixel I(x, y, t) represents the value of the pixel I(x, y) at time t. After Δt, the pixel moves by (Δx, Δy) in the next frame, which means that the grayscale value of the pixel does not change with frame tracking, as shown in Equation (6):

[0048] I(x+Δx,y+Δy,t+Δt)=I(x,y,t) (6)

[0049] Assuming that the moving distance is very small, the chain rule is used to derive formula (6), as shown in formula (7):

[0050] I x u+I y v+I t =0 (7)

[0051] Where: are the velocities in the x and y directions respectively, are the gradients of the image brightness in the x and y directions respectively. is the rate of change of image brightness over time. The local smoothness constraint assumes that the optical flow remains constant within a small local area Ω, and the overdetermined equation is shown in Equation (8):

[0052] I x (X i )u+I y (X i )v+I t (X i )=0 (8)

[0053] The optical flow estimation error is defined as shown in formula (9):

[0054]

[0055] Where Ω is a local region with n=m×m pixels, and w(x) is the weighting function of each pixel in the region, which generally follows a Gaussian distribution. The minimum value of E is obtained using the least squares method as shown in formula (10):

[0056] A T W 2 Av=A T W 2 b (10)

[0057] Where: W=diag(n(x1,…x n )). The size of v is shown in formula (11):

[0058] v=[A T W 2 A] -1 A T W 2 b (11)

[0059] Most cameras exhibit discontinuous motion. Optical flow algorithms meet the requirement of temporal continuity by constructing an image pyramid. They calculate the optical flow of an object at the top layer of the pyramid and use the result as the starting point for the next layer. This process is repeated until the optical flow of the bottom layer is reached, with the final result used as u and v. However, optical flow algorithms can result in significant positioning errors when objects move rapidly or at large angles.

[0060] Kalman filtering is divided into two steps: prediction and update. In the prediction step, the prior position of the current frame As shown in formula (12):

[0061]

[0062] Where: is the state estimate and covariance of the previous frame. u k-1 is the camera pose estimated by the optical flow algorithm, set as u k-1 =[Δx,Δy,Δz,Δθ] T , A is the state transfer matrix.

[0063] According to the Kalman filter time equation, calculate the covariance estimate of the current frame

[0064]

[0065] Where: P k-1 is the estimate of the covariance of the previous key frame. Q is the process excitation noise covariance matrix, set to a constant, and the covariance of the current frame is calculated according to the time update equation of the discrete Kalman filter

[0066] The camera motion estimated by the camera calibration and positioning algorithm is used as the observation equation:

[0067] Z k =Hx k +v k (14)

[0068] Where: Z k is the camera pose calculated by camera calibration and positioning, H is the unit matrix, v k It represents Gaussian observation noise with mean 0 and covariance R. Since the cumulative error of the optical flow algorithm is large, the covariance R of the Gaussian observation noise is set to 0.05, and the Kalman gain K k The calculation process is shown in formula (15):

[0069]

[0070] According to the state update equation of the discrete Kalman filter, the position of the entire system is updated:

[0071]

[0072]

[0073] Where: and P k They represent the pose information and covariance of the current camera after Kalman filter fusion.

[0074] This example uses an OpenMV camera to capture a video sequence following a rectangular trajectory, with the camera moving once every one second. 72 sets of continuous, unobstructed motion from the captured sequence are then selected as test samples. In the following description, each point represents a set of test results. To reduce positioning errors during the calculation process, five measurements are performed at each test point, and the average value is used as the calculation result for that point. Figure 3 Indicates the positioning results of different algorithms. The dot track represents the actual track, the dotted hollow circle track represents the positioning result of the camera calibration positioning algorithm, and the dotted asterisk represents the positioning result of the optical flow-camera calibration fusion positioning algorithm. The camera starts at the lower left corner. From 21 o'clock to 27 o'clock, an opaque plane is used to block part of the LED. From 41 o'clock, an opaque plane is used to completely block a single LED. Figure 3 It can be seen that the camera calibration positioning algorithm cannot solve the positioning after 40 points.

[0075] The actual position of the camera and the positioning error D calculated by different algorithms are shown in formula (18):

[0076]

[0077] Where: (x, y, z) represents the positioning results of different algorithms, (x r ,y r ,z r ) represents the true trajectory of the camera.

[0078] Figure 4 A relationship diagram between the positioning error and time calculated using the optical flow algorithm alone and the method of the present invention is given. From the figure, it can be seen that when the optical flow algorithm is used alone, the number of mismatched points increases after the 50th point due to large-angle rotation, resulting in a gradual increase in indoor positioning error. Figure 5 The CDF diagram of the positioning error of the method of the present invention is given. It can be seen from the diagram that the positioning error of 90% of the positioning points is within 5 cm.

[0079] To verify the robustness of the positioning method of the present invention, different experimental scenarios were set up: (1) the LED was not disturbed or blocked; (2) the LED was partially blocked; and (3) the LED was completely blocked. In most cases, the LED is intact within the camera's field of view, and the camera can capture the entire LED pixel area, achieving high-precision indoor positioning.

[0080] In the experiment, the LED began to be blocked at the 21st point. The LED could still be accurately captured when a small part of it was blocked. However, most of the LED area was blocked from the 24th to the 28th point. Since most of the LED information was lost, the results of the camera calibration and positioning algorithm deviated greatly from the actual position.

[0081] At the 40th point, an opaque plane is used to completely block one of the LEDs. At this time, the camera calibration and positioning algorithm cannot solve the positioning problem, the measurement covariance matrix error is large, the Kalman gain is small, and the weight of the predicted value in the Kalman filter output result is greater. The present invention introduces an optical flow algorithm to compensate for the result, so that the LED can still achieve high-precision indoor visible light positioning under different occlusion conditions.

[0082] In order to further verify the robustness of the method of the present invention, the maximum error value of each positioning result is calculated, and then the positioning standard deviation of the two algorithms is calculated. The results are shown in Table 2:

[0083] Table 2

[0084]

[0085] Because most of the LED information is lost at points 24 to 28, the camera calibration algorithm's results differ significantly from the actual trajectory. The camera calibration algorithm cannot determine the location after 40 points, so it only calculates the standard deviation of valid positioning points. As shown in Table 2, the positioning error and standard deviation of the proposed method are significantly smaller than those of the camera calibration algorithm. The experimental results demonstrate that the proposed method exhibits superior robustness and accuracy in low-light conditions.

[0086] In order to verify the positioning accuracy of the positioning method of the present invention under large turning angles, in the experiment, the OpenMV camera captured a video sequence according to a random movement trajectory, and the camera moved once per second. Then, 68 groups of continuous unobstructed movements were selected from the shooting sequence as test samples. The positioning results of different algorithms are shown in the figure below. Figure 6 As shown in the figure, it can be seen that the camera rotates at a large angle at the 20th point and the 49th point respectively. Figure 7 The positioning errors of different methods are shown in the figure. As can be seen from the figure, the maximum positioning error calculated by the optical flow algorithm is 13.74cm, the average positioning error is 6.86cm, and the standard deviation is 4.71cm. The maximum positioning error calculated by the optical flow-camera calibration fusion positioning algorithm is 2.76cm, the average positioning error is 1.02cm, and the standard deviation is 0.41cm. In comparison, the average positioning error calculated by the positioning method of the present invention is reduced by 85.1%, and the maximum positioning error is reduced by 79.9%. Figure 8 Given the CDF graph of positioning error, Figure 8 It can be seen that the positioning error of 90% of the positioning points is less than 1.5 cm, indicating that the positioning method of the present invention has good positioning accuracy.

[0087] To further verify the effect of adding LEDs on the camera calibration and positioning algorithm, experiments were conducted using five and six lights, with the coordinates of the two added LEDs at (130cm, 80cm, 190cm) and (70cm, 90cm, 190cm). The experimental results show that adding LEDs does not change the results of indoor visible light positioning when the camera position can be calculated.

Claims

1. The optical flow-camera calibration fusion visible light positioning method is characterized by: The following steps are involved: Step 1: Build a visible light positioning system indoors; Step 2: Use the camera to obtain image information containing white light LED at different times ; Step 3: Image information Perform preprocessing to obtain the two-dimensional coordinates of the white light LED in the pixel plane; Step 4: Calculate the camera's position in the world coordinate system using a camera calibration and positioning algorithm based on the geometric relationship between the three-dimensional coordinates of the white light LED in the world coordinate system and the two-dimensional coordinates in the pixel plane. Step 5: Determine whether the camera calibration and positioning algorithm can obtain an analytical solution. If so, use the optical flow algorithm to calculate the camera's pose, accumulate the poses calculated by the camera calibration and positioning algorithm and the optical flow algorithm, and then use the Kalman filter to perform data fusion to update the camera's pose information to complete the positioning of the visible light. If not, use the optical flow algorithm to estimate the camera's pose, combine the camera's current position, moving speed and direction, perform Kalman filter fusion, update the camera's pose information, and complete the positioning of the visible light. In step 5, the poses calculated by the camera calibration and positioning algorithm and the optical flow algorithm are accumulated, including taking the pose calculated by the optical flow algorithm as prediction information, accumulating the pose calculated by the camera calibration and positioning algorithm as update information, accumulating the accumulated three-dimensional coordinates of the camera in the world coordinate system as a translation matrix, and accumulating the three attitude angles as a rotation matrix.

2. The optical flow-camera calibration fusion visible light positioning method according to claim 1, characterized in that: In step 3, the image information Preprocessing is performed, including threshold segmentation, image denoising and centroid positioning, to obtain a binary image containing the white light LED. The two-dimensional coordinates of the white light LED in the pixel plane are calculated through the binary image.

3. The optical flow-camera calibration fusion visible light positioning method according to claim 1, characterized in that: In step 1, a visible light positioning system is built indoors, including installing a white light LED lamp, a camera, and a computer for receiving and processing camera image information indoors.

4. The optical flow-camera calibration fusion visible light positioning method according to claim 1, characterized in that: In step 2, the camera is used to obtain image information containing white light LED at different times. Before the white light LEDs are detected, the frequency of each white light LED is modulated to achieve accurate identification of different white light LEDs.

Images