Method for calibrating center catadioptric camera by straight line projection on unit sphere model

Abstract

The application is a method for calibrating the center of a linear in unit sphere model projection hyperboloid camera, characterized by using only one image containing two straight line images, five internal parameters of the camera are recovered. First, the pixel coordinates of the edge points of the two straight line images and the mirror profile image are extracted from one image respectively, and the equations of the two straight line images and the mirror profile image are obtained by using the least square method. On the basis of obtaining the equations of the two straight line images and the mirror profile image, the common vanishing point of each straight line image and the mirror profile image is obtained by solving the generalized eigenvector, and the vanishing line of the mirror profile image corresponding to the support plane is recovered by connecting the two common vanishing points. According to the polar point-polar line relationship, the polar point of the mirror profile image corresponding to the vanishing line is recovered, that is, the principal point. Because the two straight line images and the mirror profile image correspond to three different support planes, the vanishing line of the support plane corresponding to the two straight line images can be recovered by using the principal point. The vanishing line is associated with the corresponding straight line image, and the constraints of three pairs of independent circular ring point images are recovered. The image of the absolute quadratic curve is Cholesky decomposed and inversely solved, and the camera internal parameter matrix is uniquely determined.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and relates to a method for solving the internal parameters of a central catadioptric camera by using two straight lines in space and their projection properties. BACKGROUND

[0002] Computer vision, as one of the important technologies in the field of artificial intelligence, is widely used in automatic driving, image recognition and other fields. To achieve these applications, camera calibration is often needed. The central catadioptric camera combines the traditional camera with the catadioptric mirror to obtain a larger field of view, while it still retains a single effective view point. Straight lines, as common geometric shapes in the calibration scene, have good projection properties, so straight lines are often used as calibration objects to calibrate the central catadioptric camera.

[0003] Straight lines as calibration objects have two advantages: first, they are easy to obtain in daily life; second, straight lines have good projection properties in the unit sphere. The paper "Geometric properties of central catadioptric line images and their application in calibration", (Barreto J.P., Araujo H., IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1327-1333, 2005) proposed a new projection principle, which divided the traditional two-step projection into three steps: first, project from the unit sphere to the infinity plane; second, project from the infinity plane to the image plane. It was found that the projective invariance of straight line images is valid in both the infinity plane and the image plane. The paper also proposed a method for determining mirror parameters using cross ratios. This method only needs two or three straight line images to determine the relevant parameters. The paper "Using concurrent lines in central catadioptric camera calibration", (L. Zhang, D. U. Xin, J. L, Journal of Zhejiang University Science C, vol. 12, no. 3, pp: 71-81, 2011) proposed a four-step projection model: first, project from the unit sphere to the infinity plane; second, project from the infinity plane to the metric plane; third, project from the metric plane to the image plane. This method uses the projection invariance (i.e., associativity, homogeneity, and cross ratio) to calibrate the camera intrinsic parameters. The infinity plane is added to facilitate the subsequent calculation of mirror parameters; the metric plane is added to provide constraint conditions using the orthogonality of the two principal axes of the conic. This method converts the nonlinear optimization method into multiple linear problems, reducing the time complexity. However, this method requires the use of the matrix form of the mirror profile image to estimate the initial value of the camera intrinsic parameters.The document "Paracatadioptric camera calibration based on properties of polar line of infinity point with respect to circle and line", (Y. Li, Y. Zhao. International Journal of Advanced Robotic Systems, vol. 15, no. 5, 2018) uses the knowledge of topology, analyzes the properties of the corresponding topological points of the great circle formed by the straight line in the unit view sphere model, establishes the constraint of the orthogonal vanishing point, and thus obtains the camera internal parameter matrix. However, this method also needs to estimate the initial value of the internal parameter by using the image of the straight line, and the estimation may affect the accuracy of the calibration algorithm. The document "Calibrating focal length for paracatadioptric camera from one circle image", (H. Duan, L. Mei, Y. Shang. Proceedings of International Conference on Computer Vision Theory and Applications, pp: 1-8, 2014) proposes the property that the points on the same straight line are coplanar on the unit view sphere to calibrate the camera internal parameter. The biggest difference between this method and the traditional calibration method based on straight line is that it does not need to fit the straight line image. However, this calibration method has great limitations, that is, under the non-parabolic mirror surface, the time complexity of this algorithm is high, and only the focal length can be solved, and other parameters need to be estimated according to the mirror profile image. SUMMARY

[0004] The present application provides a method for calibrating a central catadioptric camera by using the image of a straight line. The method uses the properties that the optical center of the camera and the mirror profile and the great circle (the image of the straight line projected on the unit view sphere) form a right cone and an inclined cone respectively, solves the generalized eigenvalue of the right cone and the inclined cone to obtain an infinite point, and then obtains the vanishing line on the image plane from the images of the two inclined cones and the right cone, and further obtains the image of the circular ring point and the camera internal parameter.

[0005] The present application adopts the following technical scheme:

[0006] The application is a method for solving the internal parameters of a central catadioptric camera by using two straight lines in space as calibration objects, characterized in that only one image containing two straight line images is used to recover the five internal parameters of the camera. First, the pixel coordinates of the edge points of the two straight line images and the mirror profile image are extracted from the one image respectively, and the equations of the two straight line images and the mirror profile image are obtained by using the least square method.

[0007] On the basis of obtaining the equations of the two straight line images and the mirror profile image, the common vanishing points of each straight line image and the mirror profile image are obtained by solving the generalized eigenvectors, and the vanishing line of the mirror profile image corresponding to the support plane is recovered by connecting the two common vanishing points. According to the pole-polar line relationship, the pole, i.e. the principal point, of the mirror profile image corresponding to the vanishing line is recovered. Because the two straight line images correspond to three different support planes, the vanishing lines of the support planes corresponding to the two straight line images can be recovered by using the principal point. The vanishing lines are combined with the corresponding straight line images to recover the constraints of three pairs of independent circular ring point images. The image of the absolute conic is Cholesky decomposed and then inverted, and the internal parameter matrix of the camera is uniquely determined.

[0008] The specific steps include: fitting the equations of the straight line images and the mirror profile image; recovering the common vanishing points of each straight line image and the mirror profile image; determining the coordinates of the principal point and the vanishing lines of the support planes corresponding to each straight line image; and solving the internal parameters of the central catadioptric camera.

[0009] 1. Fitting the equations of the straight line images and the mirror profile image

[0010] The pixel coordinates of the edge points of the straight line images and the mirror profile image are extracted by using the Canny operator of the Edge function in the Matlab program, and the equations of the straight line images and the mirror profile image are obtained by using the least square method.

[0011] 2. Recovering the common vanishing points of each straight line image and the mirror profile image

[0012] Due to the imaging principle of the central catadioptric camera, the shooting process of the straight line actually obtains a partial curve c1 on the image plane, i.e. the image of the straight line on the image plane is c1. The internal parameter matrix of the camera is taken as where f e is the effective focal length, r is the aspect ratio, s is the skew factor, and [u0 v0 1] T is the homogeneous coordinates of the principal point of the camera. f e , r, u0, v0, and s are the five internal parameters of the camera. The pixel coordinates of the edge points of the one image containing the two straight line images and the mirror profile image are extracted by using the Canny operator of the Edge function in the Matlab program, and the quadratic curve equations of the mirror profile image and the straight line image are obtained by using the least square method. Here, c i(i = 1, 2) denote the coefficient matrix of the straight line image in the image plane, and c0 denotes the coefficient matrix of the mirror profile image in the image plane. In this paper, the same letter is used to denote the curve and its coefficient matrix for the sake of simplicity of expression.

[0013] By solving the generalized eigenvectors of a straight line image c1 and the mirror profile image c0, it is found that one of the generalized eigenvectors is the common vanishing point v1 of the straight line image c1 and the mirror profile image c0. Based on this projection property, two vanishing points v i (i = 1, 2) of the mirror profile image c0 can be obtained from the two straight line images c i (i = 1, 2), and finally the vanishing line l0 of the mirror profile image c0, i.e. the image of the infinite straight line in the plane where the mirror profile lies, can be obtained.

[0014] 3. Determining the principal point coordinates and the vanishing line of the support plane corresponding to each straight line image

[0015] After the vanishing line l0 of the mirror profile image c0 is recovered, the center o of the mirror profile image c0 is obtained according to the pole-polar line property and the known vanishing line l0, because the images of the two straight lines L1, L2 in the unit viewing sphere are two great circles C1, C2 in two different planes, so the two planes where C1 and C2 lie have two infinite straight lines L 1∞ , L 2∞ In the image plane, the images of C1 and C2 are c1 and c2 respectively, and have corresponding vanishing lines l1 and l2 respectively. Therefore, the polar lines of c1 and c2, i.e. the vanishing lines l1 and l2, can be obtained by using o. Thus, the three vanishing lines l i (i = 0, 1, 2) corresponding to c i (i = 0, 1, 2) are obtained.

[0016] 4. Solving the intrinsic parameters of the center catadioptric camera

[0017] The intersection of the mirror profile image c0 and the straight line images c i (i = 1, 2) and the corresponding vanishing lines l i (i = 0, 1, 2) recovers the constraints of a pair of absolute conics image ω. The mirror profile image c0 and the two straight line images c i (i = 1, 2) and the three corresponding vanishing lines l i (i = 0, 1, 2) provide six constraints of the absolute conics image ω, i.e. the absolute conics image ω is recovered. Finally, ω = K -T K -1 is decomposed by Cholesky and then inverted, and the intrinsic parameter matrix K is recovered, i.e. the five intrinsic parameters of the camera are obtained.

[0018] Advantages of the present application:

[0019] (1) The input information for the calibration algorithm is easy to obtain. Only one image containing two straight lines is needed to calibrate the camera's intrinsic parameters.

[0020] (2) The calibration object should be simple, and straight lines are very common in life and are very easy to extract.

[0021] (3) It is innovative in that it uses the property that the great circle and the mirror outline have a common point of infinity on the unit sphere for calibration. Attached Figure Description

[0022] Figure 1 A schematic diagram of reconstructing the ablation point and ablation line from two straight line images and a mirror contour image. Detailed Implementation

[0023] This invention provides a method for solving the intrinsic parameters of a central catadioptric camera using the projection properties of straight lines. The calibration template consists of two straight lines in space. Solving the intrinsic parameters of a central catadioptric camera using straight lines involves the following steps: extracting the edge points of the mirror contour image c0 and the two straight line images c from the image. i For the edge points (i = 1, 2), the least squares method is used to fit two straight line images c1 and c2 and a mirror profile image c0. Based on the straight line image c... i Given the equations of (i = 1, 2) and the mirror image c0, recover the image c of each straight line. i The common vanishing points v1 and v2 of the mirror image c0 and the mirror image c0 are essentially the points where the mirror image c0 and the two line images c are respectively... i (The common parts m1m2 and n1n2 of (i=1,2) are the points where the shadows disappear), such as Figure 1 Connecting the two common extinction points v1 and v2, we can recover the extinction line l0 corresponding to the supporting plane of the mirror image c0. Based on the epipolar property, we obtain the center of the mirror image c0 on the image plane, i.e., the principal point o, as shown below. Figure 1 And restore each straight line to its likeness (c). i The vanishing lines l1 and l2 corresponding to the supporting plane (i = 1, 2) are as follows: Figure 1 ), the two straight lines are like c i (i = 1, 2) and the corresponding extinction line l i By intersecting the lines (i = 1, 2), we can obtain the constraints on the images ω of two pairs of absolute conic sections. The images c of two straight lines on a single image... i (i = 1, 2) and the mirror image c0 and the corresponding extinction line l i Given six constraints on the absolute quadratic image ω (i = 0, 1, 2), the absolute quadratic image ω can be determined, and thus the intrinsic parameters K of the central catadioptric camera can be determined. The specific steps include: fitting the straight line image c... iThe equations of (i = 1, 2) and the mirror image c0; recovering the image c of each straight line. i The common vanishing point v of (i=1,2) and the mirror image c0 i (i = 1, 2); Based on the pole-epoch property and the extinction line l0, we obtain the principal point o and the image c of each straight line. i The vanishing line l corresponding to the supporting plane (i = 1, 2) i (i = 1, 2); the three disappearing lines l i (i = 0, 1, 2) and their corresponding quadratic curves c i The image m of (i = 0, 1, 2) intersects at three pairs of circular points. Ii ,m Ji Given (i = 0, 1, 2), solve for the intrinsic parameters K of the central catadioptric camera. The central catadioptric camera is calibrated using the method described in this invention, with the following specific steps:

[0024] 1. Equations for fitting straight-line images and mirror-like contour images

[0025] Extracting the line image c using the Canny operator of the Edge function in Matlab. i The pixel coordinates of the edge points of the mirror contour image c0 (i = 1, 2) are used to fit the line image c using the least squares method. i The equations for (i = 1, 2) and the mirror image c0.

[0026] 2. Restore the common null point between each straight line image and the mirror contour image.

[0027] A straight line image c1 and a mirror image c0 share a common vanishing point, which algebraically satisfies:

[0028] c0x i =l i c1x i ,(i=1,2,3). (1)

[0029] like Figure 1 , where the real number λ i (i = 1, 2, 3) are called the generalized eigenvalues ​​of the mirror image c0 and the straight image c1. This is achieved by solving for a generalized eigenvector x of the straight image c1 and the mirror image c0. i (i = 1, 2, 3), we find one of the generalized eigenvectors, denoted as x1, which is also the common vanishing point v1 of the line image c1 and the mirror contour image c0. Similarly, for the two line images c1 and c2, we can obtain the two vanishing points v1 and v2 of the mirror contour image c0, as follows. Figure 1 .

[0030] Finally, the vanishing line l0 of the mirror profile image c0 can be obtained, i.e.

[0031] v1 x v2 = l0, (2)

[0032] where "x" denotes cross product, and l0 is essentially the image of the infinite straight line of the plane where the mirror profile lies.

[0033] 3. Determining the principal point coordinates and the vanishing line of each straight line image corresponding to the support plane

[0034] After the vanishing line l0 of the mirror profile image c0 is recovered, the center o of the mirror profile image c0 can be obtained according to the pole-polar property and the known vanishing line l0, i.e. Figure 1

[0035]

[0036] Because the projections of the two straight lines L1 and L2 on the unit sphere are two great circles C1 and C2 in two different planes, the two planes where C1 and C2 lie have two infinite straight lines L 1∞ , L 2∞ In the image plane, the images of C1 and C2 are c1 and c2 respectively, and they have corresponding vanishing lines l1 and l2 respectively. Therefore, the polar lines of c1 and c2, i.e. the vanishing lines l1 and l2, can be obtained using o, i.e.

[0037]

[0038] Thus, the three vanishing lines l i (i = 0, 1, 2) corresponding to c i (i = 0, 1, 2) are obtained, i.e. Figure 1 .

[0039] 4. Solving the intrinsic parameters of the central catadioptric camera

[0040] The mirror profile image c0 and the straight line images c i (i = 1, 2) and the corresponding vanishing lines l i (i = 0, 1, 2) are intersected to recover the constraints of the image ω of the absolute conic, i.e. the equation set is obtained:

[0041]

[0042] where x is a three-dimensional column vector. Solving equation set (5) can recover the images m Ii , m Ji (i = 0, 1, 2) of the circular points, i.e. the constraints of the image ω of the absolute conic are obtained, i.e.

[0043] ​

[0044] The equation set (6) is equivalent to the following constraints on the image ω of the absolute conic:

[0045]

[0046] where Re, Im denote the real and imaginary parts of a number respectively. Let m Ii = (m i1 , m i2 , 1) T , then equation set (7) can be written as:

[0047]

[0048] where C 6×1 = [w 11 , w 12 , w 22 , w 13 , w 23 , w 33 ] T The two straight line images c i (i = 1, 2) and the mirror profile image c0 and the three corresponding vanishing lines l i (i = 0, 1, 2) provide six constraints on the image ω of the absolute conic, i.e. the least square solution of the three equation sets (8) is the image ω of the absolute conic. Finally, according to ω = K -T K -1 , Cholesky decomposition and inversion of ω are performed to recover the intrinsic parameter matrix K, i.e. the five intrinsic parameters of the camera are obtained.

[0049] Embodiment

[0050] The present application provides a method for solving the intrinsic parameters of a central catadioptric camera by using the projection property of straight lines, and a calibration template is two straight lines in space. The following describes the embodiment of the present application in detail by taking an example. The method of the present application is used to calibrate a central catadioptric camera used in an experiment, and the specific steps are as follows:

[0051] 1. Fitting the image boundary and the target curve equation

[0052] The image size used in the present application is 600 x 400. An experimental image containing two straight line images is taken by a central catadioptric camera, the pixel coordinates of the edge points of the straight line images and the mirror profile image are extracted by using the Canny operator of the Edge function in Matlab, and the equations of the straight line images and the mirror profile image are obtained by using the least square method. The coefficient matrix of the equations of the mirror profile image and the two straight line images is c i (i = 0, 1, 2), and the results are as follows:

[0053]

[0054]

[0055]

[0056] 2. Recovering the common vanishing point of each straight line image and the mirror profile image

[0057] Solving the mirror profile image c0 and the straight line image c1 with the other two different real generalized eigenvalues λ of the simultaneous equations (9) and (10) 01 = 0.765, substituting into (1), we get:

[0058] (c0- λ 01 c1) x 01 = 0, (12)

[0059] The generalized eigenvector matrix form x 01 of c0 and c1 is obtained, i.e.:

[0060] x 01 = [-1 -0.0715 -0.000000000000000045147] T , (13)

[0061] Solving the mirror profile image c0 and the straight line image c2 with the other two different real generalized eigenvalues λ of the simultaneous equations (9) and (11) 02 = 0.584, substituting into (1), we get:

[0062] (c0- λ 02 c2) x 02 = 0, (14)

[0063] The generalized eigenvector x 02 of c0 and c2 is obtained in matrix form, i.e.:

[0064] x 02 = [1 -0.343 -0.000000000000000009188] T . (15)

[0065] 3. Determining the principal point coordinates and the vanishing line of each straight line image corresponding to the support plane

[0066] Recovering x 01 and x 02 , we get the l0 matrix form, i.e.:

[0067] x 01 x x 02= l0= [-0.000000000000000035772 -0.00000000000000013109 1] T , (16)

[0068] where "x" denotes cross product, and l0is essentially the polar line of the center o with respect to the mirror profile image c0. Thus the center o is:

[0069]

[0070] In fact, the center o of the mirror profile image is the principal point. Thus we can get the polar lines of o with respect to the straight line images c1and c2, which are actually the two vanishing lines l i (i = 1, 2), in matrix form are:

[0071]

[0072] 4. Solving the intrinsic parameters of the central catadioptric camera

[0073] From equations (9)-(11), (16) and (18), the intersection of the mirror profile image c0and the straight line images c i (i = 1, 2) with the corresponding vanishing lines l i (i = 0, 1, 2) can recover the constraints of the image ω of the three pairs of absolute conics, i.e. the equations:

[0074]

[0075]

[0076]

[0077] where x is a three-dimensional column vector. The solutions of equations (19), (20) and (21) are the images m Ii , m Ji (i = 0, 1, 2) of the three pairs of different circular points, whose matrix forms are:

[0078]

[0079]

[0080]

[0081] From equations (22), (23) and (24), we can get the constraints of the image ω of the absolute conics provided by equation (6), i.e.:

[0082]

[0083] Similarly, equation set (25) is transformed and solved. The least square solution of equation set (25) is the image ω of the absolute conic, and the matrix form is:

[0084]

[0085] Finally, according to ω = K -T K -1 , Cholesky decomposition and inversion are performed on ω to restore the internal parameter matrix K, that is:

[0086]

[0087] where the aspect ratio r = K(1, 1) / K(2, 2) (where K(1, 1) represents the element of the first row and the first column of the matrix K, and K(2, 2) represents the element of the second row and the second column of the matrix K), so the five internal parameters of the camera are respectively: r = 0.94117647, f e = 850.0000, s = 0.1000, u0 = 340.0019, v0 = 260.0000.

Claims

1. A method for calibrating a central catadioptric camera with a linear projection on a unit sphere model, characterized in that Only two straight line images are used as targets; the specific steps of the method include: first, the pixel coordinates of the edge points of the two straight line images and the mirror profile image are extracted from one image respectively, and the equations of the two straight line images and the mirror profile image are obtained by using the least square method; on the basis of obtaining the equations of the two straight line images and the mirror profile image, the common vanishing points of each straight line image and the mirror profile image are obtained by solving the generalized eigenvectors, and the vanishing lines of the mirror profile image corresponding to the support planes are recovered by connecting the two common vanishing points; according to the polar point-polar line relationship, the polar point, i.e. the principal point, of the mirror profile image corresponding to the vanishing line is recovered; because the two straight line images and the mirror profile image correspond to three different support planes, the vanishing lines of the support planes corresponding to the two straight line images are recovered by using the principal point; the vanishing lines are combined with the corresponding straight line images, and the constraints of three pairs of independent circular ring point images are recovered; the image of the absolute quadratic curve is Cholesky decomposed and inversely solved, and the camera internal parameter matrix is uniquely determined; (1) Recovering the common vanishing point of each straight line image and the mirror profile image Due to the imaging principle of central reflection, the process of capturing a straight line actually obtains a partial curve c1 on the image plane; that is, the image of the straight line on the image plane is c1. The camera's intrinsic parameter matrix is ​​taken as... Where f e is the effective focal length, r is the aspect ratio, s is the tilt factor, [u0v0 1] T f is the homogeneous coordinate of the camera principal point; e r, u0, v0, s are the five intrinsic parameters of the camera; the Canny operator of the Edge function in Matlab is used to extract the pixel coordinates of the edge points of an image containing two straight line images and a mirror contour image; the quadratic curve equations of the mirror contour image and the straight line image are obtained by least squares fitting; here c i Where i = 1, 2, represents the coefficient matrix of the line image in the image plane, and c0 represents the coefficient matrix of the mirror contour image in the image plane; for simplicity, the same letters are used to represent the curve and its coefficient matrix; by solving the generalized eigenvectors of a line image c1 and a mirror contour image c0, it is found that one of the generalized eigenvectors is the common vanishing point v1 of the line image c1 and the mirror contour image c0; based on the projection property, given two line images c i This yields the two null points v of the mirror image c0. i , where i = 1, 2, and finally we get the extinction line l0 of the mirror contour image c0, which is the image of the line at infinity on the plane where the mirror contour is located. (2) Determining the principal point coordinates and the vanishing lines of the support planes corresponding to each straight line image After the shadow line l0 of the mirror profile image c0 is restored, according to the pole-polar line property, the center o of the mirror profile image c0 is obtained by using the known shadow line l0, because the images of the two straight lines L1, L2 on the unit viewing sphere are two great circles C1, C2 in two different planes, so the two planes in which C1 and C2 are located have two infinite straight lines L 1∞ , L 2∞ In the image plane, the images of C1 and C2 are c1 and c2 respectively, and have corresponding shadow lines l1 and l2 respectively; therefore, the polar lines of c1 and c2, i.e. the shadow lines l1 and l2, are obtained by using o; thus, the corresponding three shadow lines l i i i , where i=0, 1, 2. (3) Solving the internal parameters of the central catadioptric camera The mirror profile image c0and the straight line image c i where i = 1,2; and the corresponding vanishing lines l i where i = 0,1,2, are intersected to recover the constraints of the image ω of the absolute conic; the mirror profile image c0and the two straight line images c i where i = 1,2; and the corresponding vanishing lines l i where i = 0,1,2, are intersected to recover the constraints of the image ω of the absolute conic; the mirror profile image c0and the two straight line images c -T K -1 Cholesky decomposition of ω and inversion, the internal parameter matrix K is recovered, i.e. the five internal parameters of the camera are obtained.

Images