A method for calibrating the external parameters of a camera in a vehicle surround view system

Abstract

The present invention provides a method for calibrating the extrinsic parameters of a camera in a vehicle surround view system. The extrinsic parameters of the camera are calibrated using a calibration board placed non-aligned, and the world coordinates of the corner points of the calibration board do not need to be manually measured. Only four calibration boards need to be placed well to ensure that each camera can see two calibration boards at the same time. Finally, by inputting the camera installation position and the basic vehicle parameters, the extrinsic parameter calibration of the camera can be automatically completed. The present invention greatly improves the efficiency of the extrinsic parameter calibration of the camera and the intelligence of the surround view system. It can automatically calibrate the extrinsic parameters in the case of non-aligned placement of the calibration board, without manual measurement, the implementation process is simple, the degree of automation is high, and the efficiency of the extrinsic parameter calibration of the camera is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of data recognition, data representation, recording carriers, and processing of recording carriers, and particularly relates to a method for calibrating external parameters of cameras in a vehicle surround view system in the field of vehicle assisted driving. Background Art

[0002] In recent years, with the development of computer vision, panoramic images have been widely used in medical imaging technology, remote sensing image technology, virtual reality world, vehicle safety, etc. In terms of vehicle safety, the blind spots around the vehicle undoubtedly increase the potential safety hazards of the vehicle. In order to enable the driver to conveniently pay attention to the blind spot vision around the vehicle, the 360 panoramic surround view system AVM (AroundView Monitor) has emerged as the times require; the panoramic assisted driving system makes up for the deficiency of the driver's visual blind spot and can provide the driver with a 360° range of vision around the vehicle, greatly improving vehicle safety; the panoramic surround view system collects omnidirectional image information through four fisheye cameras around the vehicle, and finally generates a seamless and complete top-down image in the 360° range around the vehicle through distortion correction, perspective transformation, and fusion stitching. It is a real "blind spot terminator" and is widely used in blind spot detection, vehicle collision warning, lane departure warning, automatic parking system, etc. as a basic component of the assisted driving system.

[0003] In the panoramic surround view system, the parameters of camera calibration affect the final image stitching effect and the integrity of the panoramic image. In addition to internal parameter calibration, a commonly used method for external parameter calibration of cameras is to place aligned calibration plates around the vehicle. The alignment is to facilitate obtaining the coordinates of each corner point of the calibration plate, and at this time, the coordinates of the calibration plate corner points also need to be measured manually; although this method can obtain high-precision coordinates of the calibration plate corner points, it greatly increases the labor and time costs and is very unfavorable to the development of the panoramic surround view system. Summary of the Invention

[0004] The present invention solves the problems existing in the prior art and provides an optimized method for calibrating external parameters of cameras in a vehicle surround view system, which calibrates the camera based on non-aligned placement of the calibration plate to improve the efficiency of external parameter calibration.

[0005] The technical solution adopted by the present invention is a method for calibrating external parameters of cameras in a vehicle surround view system, and the method includes the following steps:

[0006] Step 1: Randomly place 4 calibration plates around any vehicle so that there are at least 2 calibration plates in the field of view of each camera of the vehicle surround view system; obtain the basic parameters of the vehicle;

[0007] Step 2: Select any camera C1 and a corresponding calibration plate B, and establish a calibration plate coordinate system for this calibration plate B;

[0008] Step 3: Obtain the image of camera C1. Given the coordinates of the 4 corner points of calibration board B in the calibration board coordinate system and the pixel coordinates of the 4 corner points in camera C1, calculate the external parameter matrix R' and t' of camera C1 relative to the calibration board coordinate system; obtain the coordinates of camera C1 in the calibration board coordinate system of calibration board B from R' and t'.

[0009] Step 4: For calibration board B, select another corresponding camera C2, and obtain the coordinates of camera C2 in the calibration board coordinate system of calibration board B according to the method in Step 3.

[0010] Step 5: Based on the coordinates of cameras C1 and C2 in the calibration board coordinate system of calibration board B and the coordinates of cameras C1 and C2 in the vehicle center coordinate system, obtain the conversion relationship between the calibration board coordinate system of calibration board B and the vehicle center coordinate system, and further obtain the estimated values of the coordinates of the corner points of calibration board B in the vehicle center coordinate system.

[0011] Step 6: Repeat Step 2 to Step 5 until the estimated values of the coordinates of all the corner points of the calibration boards in the vehicle center coordinate system are obtained; determine whether the difference between the estimated values of the coordinates and the pixel coordinates detected by the image exceeds the threshold. If it exceeds, optimize the corner point coordinates and the camera poses and proceed to the next step. Otherwise, directly proceed to the next step.

[0012] Step 7: According to the two calibration boards observed by each camera, calculate the external parameter R and t matrices of each camera relative to the vehicle center coordinate system.

[0013] Preferably, there are 4 calibration boards around the vehicle. There are 2 calibration boards in the field of view of any one of the cameras, and at least one of the calibration boards in the fields of view of any two cameras is different.

[0014] Preferably, the basic parameters of the vehicle include the length and width of the vehicle and the relative installation position information of each camera of the vehicle surround view system on the vehicle.

[0015] Preferably, in Step 3, the coordinates of the 4 corner points of the calibration board in the calibration board coordinate system and the pixel coordinates in the camera are used to calculate the external parameter matrix R' and t' of the camera relative to the current calibration board coordinate system through the PNP algorithm, satisfying R'P w +t' = P c , where P w represents the world coordinates of the corner point, and P c represents the camera coordinates of the corner point;

[0016] Let P c be (0, 0, 0), then P w = -R' -1 *t', P wIt is the coordinate of the current camera in the calibration board coordinate system.

[0017] Preferably, in the step 5, the conversion relationship between the calibration board coordinate system and the vehicle center coordinate system is obtained by SVD decomposition;

[0018] Let the coordinates of cameras C1 and C2 in the calibration board coordinate system be p i respectively, and in the vehicle center coordinate system be p' i respectively, and there are corresponding

[0019]

[0020] Perform SVD decomposition on the error to obtain the external parameter R = VU T , where V and U respectively represent the eigenmatrices of the singular value decomposition, and the external parameter t = p - Rp'.

[0021] Preferably, in the step 6, if the error of the estimated value of the coordinate is large, the coordinate is converted to the pixel coordinate system through the external parameter and internal parameter of each camera, the pixel coordinates in the pixel coordinate system are compared with the corner pixel coordinates detected in the image, and the corner coordinates and the camera pose are updated and optimized.

[0022] Preferably, the BA algorithm is used for optimizing the reprojection error.

[0023] Preferably, the g2o graph optimization library is used, and the side length and diagonal length of the calibration board are introduced as the constraint conditions for optimizing the reprojection error;

[0024] The vertices of g2o are the world coordinates of the four corners of the calibration board and the poses of the four cameras, and the edges of g2o are the reprojection errors of the four cameras, the side length of the calibration board, and the diagonal length of the calibration board.

[0025] Preferably, after the g2o optimization is completed, the optimized corner coordinates of the four calibration boards and the poses of the four cameras are obtained. The pose is used as the external parameter of the corresponding camera, or the pose of the camera is calculated by PNP with the corner coordinates of the calibration board and the corresponding pixel coordinates.

[0026] Preferably, the calibration board is a double-loop calibration board.

[0027] The present invention provides an optimized method for calibrating the external parameters of cameras in a vehicle surround-view system. The external parameters of the cameras are calibrated using non-aligned calibration boards, and the world coordinates of the corners of the calibration board do not need to be manually measured. Just place the four calibration boards well to ensure that each camera can see two calibration boards at the same time. Finally, input the camera installation position and the basic parameters of the vehicle to automatically complete the work of calibrating the external parameters of the cameras.

[0028] The present invention greatly improves the efficiency of camera extrinsic parameter calibration and the intelligence of the surround view system. It can automatically calibrate the extrinsic parameters without aligning the calibration board, without manual measurement, with a simple implementation process and high automation, greatly improving the efficiency of camera extrinsic parameter calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the imaging of the undistorted camera of the present invention;

[0030] Figure 2 is a schematic diagram of the radial distortion of the fisheye camera of the present invention;

[0031] Figure 3 is a schematic diagram of the reprojection error of the present invention;

[0032] Figure 4 is a schematic diagram of the extrinsic parameter calibration scenario of the present invention;

[0033] Figure 5 is a schematic diagram of the g2o optimization model of the present invention;

[0034] Figure 6 is a schematic diagram of the flow of the automatic extrinsic parameter calibration method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] The following further describes the present invention in detail with reference to embodiments, but the protection scope of the present invention is not limited thereto.

[0036] The present invention relates to a method for calibrating the extrinsic parameters of a camera in a vehicle surround view system. Different from the calibration method in the prior art, which uses an aligned calibration board and then obtains the world coordinates of each calibration board corner point through manual measurement, and combines the pixel coordinates of the calibration board corner points to obtain the rotation vector R and translation vector t of each camera relative to the world coordinate system; the method proposed by the present invention does not require manual measurement of the world coordinates of the corner points, but only needs to first estimate the world coordinates of each corner point and the R and t of each camera, and then use the reprojection error combined with the g2o library to optimize the camera pose and the three-dimensional feature point coordinates simultaneously.

[0037] In the present invention, the conversion relationship between the world coordinate system and the pixel coordinate system is

[0038]

[0039] where (X w , Y w , Z w ) is the coordinate of a certain point in the world coordinate system, R and t represent the extrinsic parameters of the camera, f represents the camera focal length, dx and dy represent the length per unit pixel in the pixel coordinate system (unit: mm), x0 and y0 represent the coordinates of the center of the camera imaging plane in the pixel coordinate system, and (u, v) represents the coordinates of this point in the pixel coordinate system.

[0040] In the present invention, in addition to conforming to the above conversion relationship, the (fisheye) camera also has a barrel distortion effect. The conversion relationship between the distorted image and the undistorted image of the fisheye image is as follows:

[0041] u d = u u (1 + k1r 2 + k2r 4 + k3r 6 )

[0042] v d = v u (1 + k1r 2 + k2r 4 + k3r 6 )

[0043] wherein, (u u , v u ) represents the coordinates of a point on the undistorted image, and (u d , v d ) represents the coordinates of this point on the distorted image; k1, k2, and k3 represent distortion coefficients, and the distortion coefficients of the fisheye camera can be obtained by the Zhang Zhengyou intrinsic parameter calibration method.

[0044] The method includes the following steps:

[0045] Step 1: Randomly place 4 calibration plates around any vehicle so that there are at least 2 calibration plates in the field of view of each camera of the vehicle surround view system; obtain the basic parameters of the vehicle;

[0046] There are 4 calibration plates around the vehicle, and 2 of them are present in the field of view of any one of the cameras. At least one of the calibration plates in the fields of view of any two cameras is different.

[0047] The basic parameters of the vehicle include the length and width of the vehicle and the relative installation position information of each camera of the vehicle surround view system on the vehicle.

[0048] The calibration plate is a checkerboard calibration plate.

[0049] Step 2: Select any camera C1 and a corresponding calibration plate B, and establish the calibration plate coordinate system of this calibration plate B;

[0050] Step 3: Obtain the image of camera C1. Given the coordinates of 4 corner points of calibration plate B in the calibration plate coordinate system and the pixel coordinates of the 4 corner points in camera C1, obtain the external parameter matrix R' and t' of camera C1 relative to the calibration plate coordinate system; obtain the coordinates of camera C1 in the calibration plate coordinate system of calibration plate B from R' and t';

[0051] In step 3, the coordinates of the calibration board coordinate system corresponding to the four corner points of the calibration board and the pixel coordinates in the camera are used to calculate the external parameter matrix R' and t' of the camera relative to the current calibration board coordinate system through the PNP algorithm, satisfying R'P w +t' = P c , where P w represents the world coordinates of the corner point, and P c represents the camera coordinates of the corner point;

[0052] Let P c be (0, 0, 0), then P w =-R' -1 *t', and P w is the coordinate of the current camera in the calibration board coordinate system.

[0053] In the present invention, since the four calibration boards are randomly placed, it is impossible to directly obtain the coordinates of all calibration board corner points in a unified coordinate system; it is necessary to establish a coordinate system with a certain corner point of each of the four calibration boards as the origin, and the coordinates of the calibration board corner points can be directly obtained in a calibration board coordinate system; at this time, taking one calibration board and one camera as a unit, knowing the world coordinates of the four corner points of the calibration board and the pixel coordinates of the corner points under this camera, the external parameter matrix R and t of the camera relative to this coordinate system can be obtained according to the above conversion method of world coordinates and pixel coordinates; finally, the coordinate of the camera in this coordinate system is obtained from R and t.

[0054] In the present invention, it is assumed that the length and width of the calibration board are 1500 mm, then the coordinates of the four corner points of the first calibration board are (0, 0, 0), (1500, 0, 0), (1500, 1500, 0) and (0, 1500, 0) in sequence, and the pixel coordinates can be obtained through manual detection or automatic detection methods; the coordinates of the calibration board coordinate system corresponding to the four corner points of the calibration board and the pixel coordinates in the camera are used to obtain the coordinates of the calibration board coordinate system and the camera coordinates through the PNP algorithm, and then the external parameter matrix R' and t' of the camera relative to the calibration board coordinate system are obtained. From R'P w +t' = P c , when P c =[0, 0, 0] T , P w =-R' -1 *t' is obtained, which is also the coordinate of the camera in the calibration board coordinate system.

[0055] Step 4: For calibration board B, select another corresponding camera C2, and obtain the coordinate of camera C2 in the calibration board coordinate system of calibration board B according to the method of step 3;

[0056] Step 5: Based on the coordinates of cameras C1 and C2 in the calibration board coordinate system of calibration board B and the coordinates of cameras C1 and C2 in the vehicle center coordinate system, obtain the conversion relationship between the calibration board coordinate system of calibration board B and the vehicle center coordinate system, and then obtain the estimated values of the coordinates of the corner points of calibration board B in the vehicle center coordinate system;

[0057] In the said Step 5, the conversion relationship between the calibration board coordinate system and the vehicle center coordinate system is obtained by SVD decomposition;

[0058] Let the coordinates of cameras C1 and C2 in the calibration board coordinate system be p i respectively, and in the vehicle center coordinate system be p' i respectively, corresponding to

[0059]

[0060] Perform SVD decomposition on the error to obtain the external parameter R = VU T , where V and U respectively represent the eigenmatrices of the singular value decomposition, and the external parameter t = p - Rp'.

[0061] In the present invention, for the coordinates of the same two points in two known coordinate systems, the method for obtaining the conversion relationship between the two coordinate systems can be regarded as a 3D-3D point set conversion, and appropriate R and t need to be obtained such that Rp' i +t ≈ p i , so the problem is transformed into a least squares problem, and SVD decomposition can be performed on the above error.

[0062] Step 6: Repeat Step 2 to Step 5 until the estimated values of the coordinates of all the corner points of the calibration board in the vehicle center coordinate system are obtained; judge whether the difference between the estimated value of the coordinate and the pixel coordinate detected in the image exceeds the threshold. If it exceeds, optimize the corner point coordinates and the camera pose and proceed to the next step. Otherwise, directly proceed to the next step;

[0063] In the said Step 6, if the error of the estimated value of the coordinate is large, the coordinate is converted to the pixel coordinate system through the external parameter and internal parameter of each camera, and the pixel coordinate of the pixel coordinate system is compared with the corner point pixel coordinate detected in the image, and the corner point coordinates and the camera pose are updated and optimized.

[0064] The optimization of the reprojection error adopts the BA algorithm

[0065] Use the g2o graph optimization library, and introduce the side length and diagonal length of the calibration board as the constraint conditions for optimizing the reprojection error;

[0066] The vertices of g2o are the world coordinates of the four corner points of the calibration board and the poses of the four cameras, and the edges of g2o are the reprojection errors of the four cameras, the side length of the calibration board, and the diagonal length of the calibration board.

[0067] After the g2o optimization is completed, the optimized corner coordinates of the four calibration boards and the four camera poses are obtained. The pose is used as the external parameter of the corresponding camera, or the corner coordinates of the calibration board and the corresponding pixel coordinates are used to calculate the camera pose through PNP.

[0068] In the present invention, in the BA algorithm, when the camera shoots the calibration board, multiple coordinate points are obtained, and the theoretical expression of their projection is

[0069] s i u i =Kexp(ξ ^ )P i

[0070] where R and t are represented by Lie algebra as exp(ξ^), s i represents the scale size, which can change the scaling degree of the image, and generally 1.0 is sufficient; K represents the camera internal parameter matrix, including the distortion coefficient, with u i representing the pixel coordinates of each point, and P i representing the world coordinates of each point;

[0071] Sum the errors of the theoretical expressions of the projections of all coordinate points, construct a least squares problem, and find the optimal camera pose and observation point coordinates to minimize this error

[0072]

[0073] where J represents the minimum value of the reprojection error, and n represents a total of n feature points.

[0074] In the present invention, since the corner coordinates of the calibration board and the camera pose are estimated values with large errors, the estimated corner coordinates are converted to the pixel coordinate system through the external and internal parameters of each camera, and the pixel coordinates are compared with the corner pixel coordinates detected in the image to update and optimize the corner coordinates and the camera pose. The BA algorithm is used to optimize the reprojection error, and the g2o graph optimization library is used for constraints to reduce the reprojection error; furthermore, according to the shape of the calibration board being square, the side length and diagonal length of the calibration board are introduced as constraint conditions, which can achieve a better optimization effect.

[0075] In the present invention, when the camera shoots this point, this point will have a pixel coordinate on the image, and the difference between this coordinate and the calculated pixel coordinates (u d , v d ) is used to update and optimize the various parameters of the camera; after obtaining multiple coordinate points, the expression of their projection is (where R and t are represented by Lie algebra as exp(ξ^)):

[0076]

[0077] Simplified to

[0078] s i u i = Kexp(ξ ^ )P i ,

[0079] Due to the unknown camera pose and noise in the observation points, there is an error in this equation; as shown in the appendix Figure 3 where P represents a feature point in the world coordinate system, and its pixel coordinates in the right pixel plane are p2, which are also the coordinates in the captured image; represents the pixel coordinates calculated by the above formula; e represents the difference between the two pixel coordinates, that is, the reprojection error of point P; in actual work, a single camera can capture multiple feature points, so the errors of all observation points can be summed to construct a least squares problem, and the optimal camera pose and observation point coordinates are found to minimize this error, from which the pose of a single camera in the world coordinate system can be obtained; and the same 3D feature point can be captured by multiple cameras, so the relationship between multiple cameras can be calculated.

[0080] Step 7: Calculate the external parameter R and t matrices of each camera relative to the vehicle center coordinate system according to the two calibration plates observed by each camera.

Claims

1. A method for calibrating the extrinsic parameters of a camera in a vehicle surround view system, characterized in that: The method includes the following steps: Step 1: Randomly place 4 calibration boards around any vehicle so that there are at least 2 calibration boards in the field of view of each camera of the vehicle surround view system; obtain the basic parameters of the vehicle; Step 2: Select any camera C1 and a corresponding calibration board B, and establish the calibration board coordinate system of this calibration board B; Step 3: Obtain the image of camera C1. Given the coordinates of 4 corner points of calibration board B in the calibration board coordinate system and the pixel coordinates of the 4 corner points in camera C1, obtain the external parameter matrix R' and t' of camera C1 relative to the calibration board coordinate system; obtain the coordinates of camera C1 in the calibration board coordinate system of calibration board B from R' and t'; Step 4: For calibration board B, select another corresponding camera C2, and obtain the coordinates of camera C2 in the calibration board coordinate system of calibration board B according to the method of Step 3; Step 5: Based on the coordinates of camera C1 and C2 in the calibration board coordinate system of calibration board B and the coordinates of camera C1 and C2 in the vehicle center coordinate system, obtain the conversion relationship between the calibration board coordinate system of calibration board B and the vehicle center coordinate system by SVD decomposition; Let the coordinates of cameras C1 and C2 in the calibration board coordinate system be p respectively i and be p' respectively in the vehicle center coordinate system i which respectively correspond to Use SVD decomposition for the error to obtain the external parameter \(R = VU\). T , where \(V\) and \(U\) respectively represent the eigenmatrices of the singular value decomposition, and the external parameter \(t = p - Rp'\), and then obtain the estimated values of the coordinates of the corner points of the calibration board \(B\) in the vehicle center coordinate system. Step 6: Repeat Step 2 to Step 5 until the estimated values of the coordinates of all corner points of the calibration boards in the vehicle center coordinate system are obtained; determine whether the difference between the estimated values of the coordinates and the pixel coordinates detected in the image exceeds the threshold. If it exceeds, optimize the corner point coordinates and the camera pose and proceed to the next step. Otherwise, directly proceed to the next step; Step 7: According to the two calibration boards observed by each camera, calculate the external parameter R and t matrices of each camera relative to the vehicle center coordinate system.

2. The method for calibrating the extrinsic parameters of a camera in a vehicle surround view system according to claim 1, wherein: There are 4 calibration boards around the vehicle, 2 of which are present in the field of view of any one of the cameras, and at least one of the calibration boards in the fields of view of any two cameras is different.

3. A method for calibrating the external parameters of a camera in a vehicle surround view system according to claim 1, characterized in that: The basic parameters of the vehicle include the length and width of the vehicle and the relative installation position information of each camera of the vehicle surround view system on the vehicle.

4. A method for calibrating the external parameters of a camera in a vehicle surround view system according to claim 1, characterized in that: In step 3, the coordinates of the calibration board coordinate system corresponding to the four corner points of the calibration board and the pixel coordinates in the camera are used to calculate the external parameter matrix R' and t' of the camera relative to the current calibration board coordinate system through the PNP algorithm, satisfying R'P w + t' = P c , where P w represents the world coordinates of the corner point, and P c represents the camera coordinates of the corner point; Let P c be (0, 0, 0), then P w = -R' -1 *t', where P w is the coordinate of the current camera in the calibration board coordinate system.

5. A method for calibrating the extrinsic parameters of a camera in a vehicle surround view system according to claim 1, characterized in that: In Step 6, if the error of the estimated value of the coordinate is large, convert the coordinate to the pixel coordinate system through the external and internal parameters of each camera, compare the pixel coordinates in the pixel coordinate system with the corner point pixel coordinates detected in the image, and update and optimize the corner point coordinates and the camera pose.

6. The method for calibrating the external parameters of a camera in a vehicle surround view system according to claim 5, wherein: The BA algorithm is used to optimize the reprojection error.

7. A method for calibrating the extrinsic parameters of a camera in a vehicle surround view system according to claim 6, characterized in that: Use the g2o graph optimization library, and introduce the side length and diagonal length of the calibration board as the constraint conditions for optimizing the reprojection error; The vertices of g2o are the world coordinates of the four corner points of the calibration board and the poses of the four cameras, and the edges of g2o are the reprojection errors of the four cameras, the side length of the calibration board, and the diagonal length of the calibration board.

8. A method for calibrating the external parameters of a camera in a vehicle surround view system according to claim 7, characterized in that: After the g2o optimization is completed, obtain the optimized corner point coordinates of the four calibration boards and the poses of the four cameras. Use the pose as the external parameter of the corresponding camera, or use the corner point coordinates of the calibration board and the corresponding pixel coordinates to calculate the pose of the camera through PNP.

9. A method for calibrating the external parameters of a camera in a vehicle surround view system according to claim 1, characterized in that: The calibration board is a double-loop calibration board.

Images