A camera pose calibration method based on a single vanishing point

Abstract

This invention proposes a camera pose calibration method based on a single vanishing point, belonging to the fields of computer vision, photogrammetry, and SLAM technology. The method utilizes only a single vanishing point generated by parallel lines of known orientation in the field of view, measures the roll angle, captures an image of the parallel lines, and obtains the image of the parallel lines within the image. The pixel coordinates corresponding to the vanishing point are then calculated to calibrate the camera pose. The proposed method utilizes only a single vanishing point and a known camera roll angle to independently determine the camera pose, reducing the number of required vanishing points and eliminating the need for a known camera focal length. This improves computational speed and provides a unique solution.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, photogrammetry and SLAM technology, and particularly relates to a camera attitude calibration method based on a single vanishing point. Background Art

[0002] In computer vision, photogrammetry, SLAM, and other related fields, the camera's pose in the world coordinate system needs to be calibrated for measurement and estimation. Currently, there are many pose calibration methods, such as the feature point-based pose calibration method [A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation], known as the PnP algorithm, when the camera's intrinsic parameters are known or ideal intrinsic parameters can be used; and the feature line-based pose calibration method [A novel algebraic solution to the perspective-three-line pose problem], known as the PnL algorithm. With technological advancements, pose sensors, such as IMUs, are widely used, such as in smartphones. Furthermore, the accuracy of inexpensive IMUs is increasing, and the devices are becoming increasingly smaller and cheaper. As a result, in fields such as computer vision, more and more cameras are equipped with pose sensors to obtain the camera's roll angle, which simplifies their pose calibration. The present invention calibrates the camera's pose based on a single vanishing point, when the camera's roll angle is known.

[0003] In the PnP algorithm, if the 3D point positions and corresponding image mappings are known, various methods can be used to solve the camera pose. However, most methods cannot solve either the pose or the position independently during the pose solution process, so the number of 3D points cannot be reduced. When the number of 3D points with known positions is 3, up to 4 solutions can be obtained through nonlinear pose solution, and the corresponding algorithm is called P3P. As the number of 3D points increases, some camera intrinsic parameters can be solved simultaneously through nonlinear pose solution, such as P4Pf and P5Pfr. When the number of 3D points with known positions is ≥6, the camera position can be solved linearly, which is called the DLT method.

[0004] However, in practical applications of PnP algorithms, obtaining 3D points with known locations is a difficult task. In some scenes, 3D lines are numerous, such as indoor ceilings and outdoor building walls. Consequently, a number of PnL algorithms have been proposed. Like PnP methods, the representation of 3D lines in the world coordinate system is sometimes difficult to determine, thus limiting the applicability of PnL. In computer vision, numerous parallel lines exist, such as indoor floor tiles, ceiling gaps, and horizontal and vertical corners of buildings. These parallel lines often intersect at vanishing points in an image. Numerous methods have been proposed to calculate camera pose using vanishing points. The paper [Camera calibration combining images with two vanishing points] proposes a method for calculating camera pose using two vanishing points. However, this method requires that the two sets of parallel lines corresponding to the two vanishing points be perpendicular to each other, with the intersection of the two sets of parallel lines being the origin of the world coordinate system. Increasing the number of vanishing points or adding other constraints (such as a known length of a line segment) can address these shortcomings, while also increasing the number of computable parameters. However, the number of vanishing points also limits the algorithm's applicability, as not all scenes have many vanishing points, and the corresponding parallel lines' spatial positions in the world coordinate system are not always known. Furthermore, when solving for camera pose using only a few features, most methods require known camera intrinsic parameters, such as focal length.

[0005] To address the above issues, we have developed a fast method for calculating camera pose using a single vanishing point. This method eliminates the need for knowing the camera's focal length when calculating pose. By leveraging the roll angle provided by the IMU, we only need a single vanishing point to calculate the camera pose. Furthermore, the parallel line corresponding to this vanishing point does not require a specific spatial position; only its direction is sufficient. This method is suitable for applications where the camera's roll angle can be determined in advance, parallel lines exist in the field of view, and the number of available vanishing points is small. Summary of the Invention

[0006] The purpose of this invention is to propose a camera pose calibration method based on a single vanishing point, which solves the technical problem in the prior art that when solving the camera pose and position simultaneously, ≥3 vanishing points or landmark points are required, or other constraints need to be added when the number is less than 3.

[0007] In order to achieve the above objectives and solve the above technical problems, the technical solutions of the present invention are as follows:

[0008] A camera pose calibration method based on a single vanishing point includes the following steps:

[0009] S1 obtains the relationship between the vanishing point pixel coordinates and the straight line direction vector

[0010] Assume that the direction vector of a straight line L1 in the space of the camera coordinate system is d=(dx d y d z ), and the straight line passes through a point P1(p 1x p 1y p 1z )

[0011] Then the straight line L1 can be expressed as

[0012] L1=P1+k1·d (1)

[0013] Where k1 is the scale factor, which is an arbitrary value; similarly, passing through another point P2 (p 2x p 2y p 2z ), and the straight line parallel to L1 is expressed as L2 = P2 + k2·d;

[0014] According to the pinhole camera imaging principle, the image of the point on the straight line L1 in the image is

[0015]

[0016] Here, (u1, v1) is the pixel coordinate of the point on the line L1 in the image. When L1 tends to infinity and d z When both are not zero, the image of the infinite point on the straight line is

[0017]

[0018] Similarly, the imaging of the L2 infinity point can be obtained as

[0019]

[0020] Here, (u2, v2) is the pixel coordinate of the point on the line L2 in the image. The two parallel lines intersect at infinity at the same point, which is the vanishing point.

[0021]

[0022] In the image, the image pixel coordinates of the vanishing point (u vp v vp );

[0023] S2 uses the vanishing point to calculate the initial camera pose

[0024] The direction vectors of two parallel lines are

[0025] d=(d x d y d z )=d z ·(u vp v vp1) (6)

[0026] d z is not zero, so the unit direction vector of the parallel line is

[0027]

[0028] Calculate the unit direction vector of the parallel line based on the image pixel coordinates of the vanishing point;

[0029] According to the conversion relationship between the world coordinate system and the camera coordinate system, the relationship between the unit vector on the line L1 in the two coordinate systems is obtained as follows:

[0030]

[0031] Among them, d w =(d wx d wy d wz ) is the unit vector of the parallel line in the world coordinate system; R c_w Is the rotation matrix, including the three posture information of the camera (a x a y a z ), use the sensor to set the camera roll angle to zero, then a y =0, the other two attitude angles are the quantities to be determined;

[0032] According to the rigid body rotation, we get

[0033]

[0034] When a set of parallel line unit direction vectors d in the world coordinate system w =(d wx d wy d wz ) is known, then according to its vanishing point and formula (8), three equations are obtained, among which the third equation is

[0035] d cy ·sina x +d cz ·cosa x =d wz (10)

[0036] but

[0037]

[0038] According to a x And the first equation of formula (8), we get a z ,as follows

[0039]

[0040] This allows the camera pose to be calibrated using the vanishing point.

[0041] Furthermore, in order to improve the accuracy of camera posture calibration, step S3 is also included.

[0042] S3 optimizes camera posture

[0043] Since the formula (8) contains 3 equations and 2 unknowns, the a x , a z As the initial value, record it as (a x0 a z0 ), and then use the nonlinear optimization method to get the final camera pose, as follows:

[0044] According to formulas (8)-(9), the cost function is established

[0045]

[0046] The goal is to obtain the value of the initial value (a x0 a z0 ) Perform nonlinear optimization to find the optimal solution;

[0047] The goal of posture optimization is to i (i=1,2,3) is minimized; according to the cost function, the Jacobian matrix J is obtained as

[0048]

[0049] Then use formula (15) to update the camera pose

[0050]

[0051] Formula (15) is an iterative formula. A parameter t is established to stop the iteration, as follows:

[0052]

[0053] When t is lower than the set threshold, the iteration stops and the camera pose optimization is completed.

[0054] Furthermore, the nonlinear optimization method is the LM method.

[0055] Compared with the prior art, the effective benefits of the present invention are as follows:

[0056] 1. The method proposed in this invention solves the technical problem that existing algorithms require ≥3 vanishing points or landmark points when solving the camera pose and position at the same time, or need to add other constraints when the number is less than 3.

[0057] 2. The method proposed in the present invention can independently solve the camera pose using only one vanishing point and a known camera roll angle, reducing the number of required vanishing points and not requiring a known camera focal length, thereby increasing the calculation speed and providing a unique solution.

[0058] 3. The invention is suitable for application scenarios where the camera roll angle can be obtained in advance, the number of available vanishing points in the field of view is small, and the camera focal length is difficult to obtain (such as with a zoom lens). BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of parallel lines and vanishing point imaging;

[0060] Figure 2 Parallel line imaging diagram. DETAILED DESCRIPTION

[0061] The present invention will be explained and illustrated in detail below with reference to the accompanying drawings and specific embodiments.

[0062] The implementation concept of the present invention is as follows: The present invention utilizes parallel lines with known directions and their corresponding vanishing points in the image, and then uses sensors to obtain the camera roll angle to calibrate the camera pose. This method separates pose and position parameters, eliminating coupling and enabling independent pose calculation without requiring a known camera focal length. This significantly improves computational speed and the algorithm's applicability. By utilizing the high-precision roll angle information provided by the sensor, the present invention simplifies the pose rotation matrix in the camera model, allowing the remaining two orientation quantities to be solved independently during the calculation process, significantly improving efficiency and accuracy while providing a unique solution. Next, a vanishing point is obtained using parallel lines with known directions. Parallel lines with known directions are numerous in the real world, such as roads, floor tiles, and ceilings on horizontal surfaces. The vanishing point captures the pose rotation relationship between the camera coordinate system and the world coordinate system, without involving translation, allowing for independent camera pose calibration. After determining the initial camera pose, the present invention establishes a nonlinear optimization method to optimize the pose, improving the calibration accuracy of the present invention.

[0063] This method uses only a single vanishing point generated by parallel lines of known orientation in the field of view to measure the roll angle, capture an image of the parallel lines, and then obtain the pixel coordinates corresponding to the vanishing point to calibrate the camera pose. Compared to existing calibration methods based on multiple vanishing points or feature points, this method reduces the number of vanishing points, shortens calibration time, and provides a unique solution.

[0064] (1) Calculate the initial camera pose using the vanishing point. Assume that the direction vector of a straight line L1 in the camera coordinate system is d = (d x d y d z ), and the straight line passes through a point P1(p 1xp 1y p 1z ),like Figure 1 shown.

[0065] Then the straight line L1 can be expressed as

[0066] L1=P1+k1·d (1)

[0067] Where k1 is the scale factor, any value. Similarly, passing through another point P2 (p 2x p 2y p 2z ), and the straight line parallel to L1 can be expressed as L2 = P2 + k2·d. According to the pinhole camera imaging principle, the image of the point on the straight line L1 in the image is

[0068]

[0069] When L1 tends to infinity and d z When both are not zero, the image of the infinite point on the straight line is

[0070]

[0071] Similarly, the imaging of the L2 infinity point can be obtained as

[0072]

[0073] It can be seen that the infinity points of two parallel lines intersect at the same point, which is the vanishing point

[0074]

[0075] In the image, the image coordinates of the vanishing point can be obtained by feature extraction, that is (u vp v vp ) is a known quantity. Then the direction vectors of these two parallel lines are

[0076] d=(d x d y d z )=d z ·(u vp v vp 1) (6)

[0077] d z is not zero, so the unit direction vector of the parallel line is

[0078]

[0079] It can be seen that according to the vanishing point, the unit direction vector of the parallel line can be calculated. According to the conversion relationship between the world coordinate system and the camera coordinate system, the relationship between the unit vector on the straight line L1 in the two coordinate systems is obtained as follows:

[0080]

[0081] Among them, d w =(d wx d wy d wz ) is the unit vector of the parallel line in the world coordinate system; R c_w Is the rotation matrix, including the three posture information of the camera (a x a y a z ), use the sensor to set the camera roll angle to zero, then a y = 0, and the other two attitude angles are the quantities to be determined. According to the rotation of the rigid body, we can get

[0082]

[0083] When a set of parallel line unit direction vectors d in the world coordinate system w =(d wx d wy d wz ) is known, then according to its vanishing point and formula (8), three equations can be obtained. Among them, the third equation is

[0084] d cy ·sina x +d cz ·cosa x =d wz (10)

[0085] but

[0086]

[0087] According to a x And the first equation of formula (8), we get a z ,as follows

[0088]

[0089] This allows the camera pose to be calibrated using the vanishing point.

[0090] In addition, to improve the accuracy of camera pose calibration, the following methods can be used to optimize the camera pose.

[0091] (2) Optimize the camera posture. Since formula (8) contains 3 equations and 2 unknowns, it is over-constrained. x, a z As the initial value, record it as (a x0 a z0 ), and then a nonlinear optimization method, such as the LM method, is used to obtain the final camera pose, as follows.

[0092] According to formulas (8)-(9), the cost function is established

[0093]

[0094] The goal is to get from equation (12) that the initial value (a x0 a z0 ) to perform nonlinear optimization to find the optimal solution.

[0095] The goal of posture optimization is to i (i=1,2,3), and minimize. According to the cost function, the Jacobian matrix J is obtained as

[0096]

[0097] Then update the camera pose using the following equation

[0098]

[0099] The above is an iterative formula. Here we establish a parameter to stop the iteration, as follows

[0100]

[0101] When t is lower than the set threshold, the iteration stops and the pose optimization is completed.

[0102] Example

[0103] Given two parallel lines, their direction vectors in the world coordinate system are (0, 10, 0), passing through the points (0, 0, 75) and (10, 0, 80) respectively. The camera is placed at (50, -50, 0), with a focal length of 20mm and a pixel size of 14μm. The three true angle values ​​of the camera are (-45°, 0°, 45°). The images of the two parallel lines in the camera are as follows: Figure 2 shown.

[0104] A 0.1 pixel error is added to the vanishing point calculation. Based on the imaging of parallel lines, the pixel coordinates of the corresponding vanishing point are (2011.2, 1420.7). Using the method described in this invention, the camera attitude angles are calculated to be (-45.16°, 0°, 44.95°). Except for the roll angle, the errors in the other two attitude angles are 0.16° and 0.05°, respectively. As can be seen, the attitude angle errors are very small, demonstrating the high calculation accuracy of the method described in this invention.

Claims

1. A camera pose calibration method based on a single vanishing point, characterized in that: The steps include: S1 obtains the relationship between the vanishing point pixel coordinates and the straight line direction vector Assume that the direction vector of a straight line L1 in the space of the camera coordinate system is d=(d x d y d z ), and the straight line passes through a point P1(p 1x p 1y p 1z ) Then the straight line L1 can be expressed as L1=P1+k1·d (1) Where k1 is the scale factor, which is an arbitrary value; similarly, passing through another point P2 (p 2x p 2y p 2z ), and the straight line parallel to L1 is expressed as L2 = P2 + k2·d; According to the pinhole camera imaging principle, the image of the point on the straight line L1 in the image is Here, (u1, v1) is the pixel coordinate of the point on the line L1 in the image. When L1 tends to infinity and d z When both are not zero, the image of the infinite point on the straight line is Similarly, the imaging of the L2 infinity point can be obtained as Here, (u2, v2) is the pixel coordinate of the point on the line L2 in the image. The two parallel lines intersect at infinity at the same point, which is the vanishing point. In the image, the image pixel coordinates of the vanishing point (u vp v vp ); S2 uses the vanishing point to calculate the initial camera pose The direction vectors of the two parallel lines are d=(d x d y d z )=d z ·(u vp v vp 1) (6) d z is not zero, so the unit direction vector of the parallel line is Calculate the unit direction vector of the parallel line based on the image pixel coordinates of the vanishing point; According to the conversion relationship between the world coordinate system and the camera coordinate system, the relationship between the unit vector on the line L1 in the two coordinate systems is obtained as follows: Among them, d w =(d wx d wy d wz ) is the unit vector of the parallel line in the world coordinate system; R c_w Is the rotation matrix, including the three posture information of the camera (a x a y a z ), use the sensor to set the camera roll angle to zero, then a y =0, the other two attitude angles are the quantities to be determined; According to the rigid body rotation, we get When a set of parallel line unit direction vectors d in the world coordinate system w =(d wx d wy d wz ) is known, then according to its vanishing point and formula (8), three equations are obtained, among which the third equation is d cy ·sina x +d cz ·What x =d wz (10) but According to a x And the first equation of formula (8), we get a z ,as follows This allows the camera pose to be calibrated using the vanishing point.

2. The camera pose calibration method based on a single vanishing point according to claim 1, characterized in that: To improve the accuracy of camera attitude calibration, step S3 is also included; S3 optimizes camera posture Since the formula (8) contains 3 equations and 2 unknowns, the a x , a z As the initial value, record it as (a x0 a z0 ), and then use the nonlinear optimization method to get the final camera pose, as follows: According to formulas (8)-(9), the cost function is established The goal is to obtain the value of the initial value (a x0 a z0 ) Perform nonlinear optimization to find the optimal solution; The goal of posture optimization is to i (i=1,2,3) is minimized; according to the cost function, the Jacobian matrix J is obtained as Then use formula (15) to update the camera pose Formula (15) is an iterative formula. A parameter t is established to stop the iteration, as follows: When t is lower than the set threshold, the iteration stops and the camera pose optimization is completed.

3. The camera pose calibration method based on a single vanishing point according to claim 2, characterized in that: The nonlinear optimization method is the LM method.

Images