Camera calibration method and device, equipment and storage medium

Abstract

The invention discloses a camera calibration method, device and equipment and a storage medium. A multi-target calibration device carrying a plurality of calibration cameras is used for realizing external parameter calibration of a to-be-calibrated camera of a vehicle; based on a moving trajectory method, obtaining relative external parameters between calibration cameras on the multi-target calibration device; the multi-target calibration device is arranged on a vehicle, and a to-be-calibrated camera on the vehicle and a calibration camera on the multi-target calibration device have a common-view area; based on the relative external parameters between the calibration cameras, adopting pnp projection and epipolar constraint optimization to obtain relative external parameters between each to-be-calibrated camera on the vehicle and any calibration camera in the multi-target calibration device; and based on the relative external parameter between each to-be-calibrated camera and any calibration camera in the multi-target calibration device, adopting a factor graph optimization method to obtain the external parameter of each to-be-calibrated camera relative to the automobile body coordinate system base link. According to the invention, multi-camera joint calibration which is high in precision, rapid, capable of being calibrated outdoors and high in generalization can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera calibration, and in particular relates to a camera calibration method, device, equipment and storage medium. Background Art

[0002] Camera calibration is one of the key links in autonomous driving vehicles. The calibration accuracy directly determines the performance of the autonomous driving algorithm and can ensure the driving safety of autonomous driving vehicles.

[0003] Existing calibration methods have poor accuracy, rely on initial values ​​and calibration intervals, and can only calibrate between two cameras. They are unable to achieve fast, robust, outdoor, and high-precision calibration. Incorrect calibration results will lead to a decline in the performance of the multimodal fusion algorithm. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a camera calibration method, device, equipment and storage medium.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention discloses a camera calibration method, which uses a multi-target calibration device equipped with multiple calibration cameras to implement extrinsic calibration of a camera to be calibrated of a vehicle;

[0007] Camera calibration methods include:

[0008] Step S1: Based on the running trajectory method, the relative external parameters between each calibration camera on the multi-target calibration device are obtained;

[0009] Step S2: placing a multi-object calibration device on a vehicle, and the camera to be calibrated on the vehicle and the calibration camera on the multi-object calibration device have a common viewing area;

[0010] Based on the relative extrinsic parameters between each calibrated camera, the relative extrinsic parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device are obtained by using pnp projection and epipolar constraint optimization.

[0011] Step S3: Based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device, the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink are obtained by using a factor graph optimization method.

[0012] Based on the above technical solution, the following improvements can be made:

[0013] As a preferred solution, step S1 includes:

[0014] Step S1.1: The main body of the multi-target calibration device carries multiple calibration cameras and moves, and at each point of the movement, each calibration camera is aligned with the chessboard in turn to capture images;

[0015] Step S1.2: obtaining the running trajectory of each calibration camera of the multi-target calibration device according to the images taken by each calibration camera;

[0016] Step S1.3: obtaining the running track of the device body of the multi-target positioning device according to the moving points of the multi-target positioning device;

[0017] Step S1.4: based on the running trajectory of the device body and the running trajectory of each calibration camera, obtaining the external parameters of each calibration camera relative to the device body;

[0018] Step S1.5: Based on the external parameters of each calibration camera relative to the device body, obtain the relative external parameters between each calibration camera.

[0019] As a preferred solution, step S2 includes:

[0020] Step S2.1: a multi-object calibration device is placed on a vehicle, and each camera to be calibrated on the vehicle forms a camera group with several calibration cameras on the multi-object calibration device, and the cameras to be calibrated and the calibration cameras in each camera group can observe the same chessboard grid together;

[0021] Step S2.2: performing pnp projection and epipolar constraint optimization on the camera to be calibrated and the calibration camera in each camera group to obtain relative external parameters between the cameras in each camera group;

[0022] Step S2.3: Based on the extrinsic parameters between the cameras in each camera group and the relative extrinsic parameters between each calibration camera, the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-object calibration device are obtained.

[0023] As a preferred solution, step S3 includes:

[0024] Step S3.1: Obtain the translation measurement value of each camera to be calibrated on the vehicle in the vehicle body coordinate system baselink;

[0025] Step S3.2: Based on the translation measurement value of each camera to be calibrated in the vehicle body coordinate system baselink, the factor graph optimization method is used to optimize the external parameters of any calibration camera in the multi-target calibration device in the vehicle body coordinate system baselink;

[0026] Step S3.3: Based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device and the external parameters of the optimized calibration camera in the vehicle body coordinate system baselink, obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink.

[0027] In a second aspect, the present invention discloses a camera calibration device, which uses a multi-target calibration device equipped with multiple calibration cameras to implement external parameter calibration of a camera to be calibrated of a vehicle;

[0028] The camera calibration device includes:

[0029] The hand-eye calibration module is used to obtain the relative external parameters between each calibration camera on the multi-target calibration device based on the running trajectory method;

[0030] A joint calibration module is used to place a multi-object calibration device on a vehicle, and the camera to be calibrated on the vehicle and the calibration camera on the multi-object calibration device have a common viewing area;

[0031] Based on the relative extrinsic parameters between each calibrated camera, the relative extrinsic parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device are obtained by using pnp projection and epipolar constraint optimization.

[0032] The factor graph optimization module is used to obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink by using the factor graph optimization method based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device.

[0033] As a preferred solution, the hand-eye calibration module includes:

[0034] A motion unit, a device body for a multi-target calibration device, carries multiple calibration cameras for movement, and at each point of the movement, each calibration camera is aligned with the chessboard in turn to capture images;

[0035] A first running trajectory determining unit, used for obtaining the running trajectory of each calibration camera of the multi-target calibration device according to the image taken by each calibration camera;

[0036] A second running track determination unit is used to obtain the running track of the device body of the multi-target positioning device according to the motion point position of the multi-target positioning device;

[0037] A first external parameter determination unit, configured to obtain an external parameter of each calibration camera relative to the device body based on the running trajectory of the device body and the running trajectory of each calibration camera;

[0038] The second extrinsic parameter determination unit obtains relative extrinsic parameters between each calibration camera based on the extrinsic parameters of each calibration camera relative to the device body.

[0039] As a preferred solution, the joint calibration module includes:

[0040] A placement unit is used to place the multi-object calibration device on the vehicle, and each camera to be calibrated on the vehicle forms a camera group with a plurality of calibration cameras on the multi-object calibration device, and the camera to be calibrated and the calibration camera in each camera group can observe the same chessboard grid together;

[0041] A third extrinsic parameter determination unit is used to perform pnp projection and epipolar constraint optimization on the camera to be calibrated and the calibration camera in each camera group to obtain relative extrinsic parameters between the cameras in each camera group;

[0042] The fourth extrinsic parameter determination unit is used to obtain the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device based on the extrinsic parameters between each camera in each camera group and the relative extrinsic parameters between each calibration camera.

[0043] As a preferred solution, the factor graph optimization module includes:

[0044] A translation measurement value acquisition unit, used to acquire the translation measurement value of each camera to be calibrated on the vehicle in the vehicle body coordinate system baselink;

[0045] A factor graph optimization unit is used to optimize the external parameters of any calibration camera in the multi-target calibration device in the body coordinate system baselink based on the translation measurement value of each camera to be calibrated in the body coordinate system baselink by using a factor graph optimization method;

[0046] The fifth extrinsic parameter determination unit is used to obtain the extrinsic parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink based on the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device and the extrinsic parameters of the optimized calibration camera in the vehicle body coordinate system baselink.

[0047] In a third aspect, the present invention discloses a computing device, comprising:

[0048] one or more processors;

[0049] Memory;

[0050] and one or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs include instructions of any of the above-mentioned camera calibration methods.

[0051] In a fourth aspect, the present invention discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned camera calibration methods.

[0052] The present invention discloses a camera calibration method, device, equipment and storage medium, which have the following

[0053] Beneficial effects:

[0054] First, the present invention discloses a multi-target positioning device, which has a simple structure and can obtain the relative external parameters between each calibration camera on the multi-target positioning device by analyzing the running trajectory. The multi-target positioning device can be applied to camera calibration occasions.

[0055] Second, the present invention uses a multi-target calibration device to calibrate the camera on the vehicle, and adopts pnp projection and epipolar constraint optimization to obtain the relative external parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device.

[0056] Third, the present invention adopts a factor graph optimization method to obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink.

[0057] Fourth, the present invention can achieve high-precision, fast, outdoor-calibration, and highly generalized multi-camera joint calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A schematic diagram of a multi-target positioning device provided in an embodiment of the present invention.

[0060] Figure 2 A flowchart of a camera calibration method provided by an embodiment of the present invention.

[0061] Figure 3 A schematic diagram of the running trajectory of the calibration camera and the running trajectory of the device body provided in an embodiment of the present invention.

[0062] Figure 4 A schematic diagram of a vehicle and a multi-target positioning device provided in an embodiment of the present invention.

[0063] Figure 5 A schematic diagram of a factor graph provided by an embodiment of the present invention.

[0064] Figure 6 A schematic diagram of binocular stereo correction results provided by an embodiment of the present invention.

[0065] Figure 7A schematic diagram of the projection results of the panoramic camera and laser radar provided in an embodiment of the present invention.

[0066] Figure 8 A schematic diagram of the projection results of the surround view camera and lidar provided in an embodiment of the present invention.

[0067] Fig. 9 A block diagram of a camera calibration device provided in an embodiment of the present invention.

[0068] Fig.10 A block diagram of a computing device provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0072] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0073] In order to achieve the purpose of the present invention, in some embodiments of the camera calibration method, a multi-target calibration device equipped with multiple calibration cameras is used to implement extrinsic calibration of the camera to be calibrated of the vehicle.

[0074] like Figure 1 As shown, an eight-target calibration device equipped with eight calibration cameras is taken as an example below. Of course, the present invention is not limited to the eight-target calibration device, and a device with other numbers of calibration cameras is also acceptable.

[0075] The eight-target calibration device disclosed in the present invention has eight calibration cameras: CAM-A0, CAM-B0, CAM-C0, CAM-D0, CAM-E0, CAM-F0, CAM-G0, and CAM-H0.

[0076] Among them: CAM-A0, CAM-B0, CAM-C0, and CAM-D0 are the main calibration cameras, which are set at the four directions of front, back, left, and right respectively;

[0077] CAM-E0, CAM-F0, CAM-G0, and CAM-H0 are backup calibration cameras, which are backup redundancy and are set at the four corner points of the left front, right front, left rear, and right rear respectively.

[0078] The main functions of backup redundancy are: 1) When the main calibration camera is damaged, the backup calibration camera can be used; 2) When there are additional camera groups that need to be calibrated in the left front, right front, left rear or right rear direction of the vehicle, the backup camera can be used.

[0079] Specifically, Figure 2 As shown, the camera calibration method includes:

[0080] Step S101: Based on the running trajectory method, the relative external parameters between each calibration camera on the multi-target calibration device are obtained;

[0081] Step S102: placing a multi-object calibration device on a vehicle, and the camera to be calibrated on the vehicle and the calibration camera on the multi-object calibration device have a common viewing area;

[0082] Based on the relative extrinsic parameters between each calibrated camera, the relative extrinsic parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device are obtained by using pnp projection and epipolar constraint optimization.

[0083] Step S103: Based on the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device, a factor graph optimization method is used to obtain the extrinsic parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink.

[0084] Each step is described in detail below.

[0085] In the specific operation, step S101 can be completed with the help of a robotic arm. The multi-target positioning device is placed on the robotic arm.

[0086] Further, step S101 includes:

[0087] Step S101.1: The robot arm drives the device body of the multi-target calibration device to carry multiple calibration cameras to move, and at each point of the movement, each calibration camera is aligned with the chessboard in turn to capture an image;

[0088] Step S101.2: obtaining the running trajectory of each calibration camera of the multi-target calibration device according to the images captured by each calibration camera;

[0089] Here, the SFM (Structure from motion) method can be used to solve the trajectory of each calibrated camera;

[0090] Step S101.3: obtaining the running track of the device body of the multi-target positioning device according to the moving points of the multi-target positioning device;

[0091] It is worth noting that the running trajectory of the device body here is the running trajectory of the end effector of the robot arm;

[0092] Step S101.4: based on the running trajectory of the device body and the running trajectory of each calibration camera, obtaining the external parameters of each calibration camera relative to the device body;

[0093] Step S101.5: Based on the external parameters of each calibration camera relative to the device body, obtain the relative external parameters between each calibration camera.

[0094] Step S101.4 is specifically as follows:

[0095] like Figure 3 As shown, W is the reference coordinate system, C0 is the starting position of any calibrated camera, and C i For any calibration camera endpoint pose, C0C i is the running trajectory of any calibration camera, A0 is the starting position of the device, A i is the final position of the device, A0A i It is the running track of the device body.

[0096] Note the posture transformation relationship:

[0097]

[0098] in,

[0099] is the pose of A0 in the reference coordinate system W;

[0100] It is C i The pose relative to C0;

[0101] Yes A i The pose relative to A0;

[0102] T AC It is the external parameter of the calibration camera relative to the device body.

[0103] Define external parameter T:

[0104]

[0105] Among them, R is the rotation and t is the translation.

[0106] Define the hand-eye calibration residual function:

[0107]

[0108] Minimize the residual function of hand-eye calibration, that is, get T AC .

[0109] Through the above content, the above hand-eye calibration can be performed on the calibration cameras CAM-A0, CAM-B0, CAM-C0, CAM-D0, CAM-E0, CAM-F0, CAM-G0, and CAM-H0, and the external parameter T of each calibration camera relative to the device body can be obtained. _A_CAM-A0 , T _A_CAM-B0 , T _A_CAM-C0 , T _A_CAM-D0 , T _A_CAM-E0 , T _A_CAM-F0 , T _A_CAM-G0 , T _A_CAM-H0 .

[0110] Among them, T _A_CAM-A0 It is the external parameter of the calibration camera CAM-A0 relative to the device body;

[0111] T _A_CAM-B0 It is the external parameter of the calibration camera CAM-B0 relative to the device body;

[0112] T _A_CAM-C0 It is the external parameter of the calibration camera CAM-C0 relative to the device body;

[0113] T _A_CAM-D0 It is the external parameter of the calibration camera CAM-D0 relative to the device body;

[0114] T _A_CAM-E0 It is the external parameter of the calibration camera CAM-E0 relative to the device body;

[0115] T _A_CAM-F0 It is the external parameter of the calibration camera CAM-F0 relative to the device body;

[0116] T _A_CAM-G0 It is the external parameter of the calibration camera CAM-G0 relative to the device body;

[0117] T _A_CAM-H0 It is the external parameter of the calibration camera CAM-H0 relative to the device body.

[0118] Furthermore, the relative external parameters between each calibrated camera can be obtained by the following formula:

[0119] T _CAM-A0_CAM-A0 =T _A_CAM-A0 -1 T_A_CAM-A0

[0120] T _CAM-A0_CAM-B0 =T _A_CAM-A0 -1 T _A_CAM-B0

[0121] T _CAM-A0_CAM-C0 =T _A_CAM-A0 -1 T _A_CAM-C0

[0122] T _CAM-A0_CAM-D0 =T _A_CAM-A0 -1 T _A_CAM-D0

[0123] T _CAM-A0_CAM-E0 =T _A_CAM-A0 -1 T _A_CAM-E0

[0124] T _CAM-A0_CAM-F0 =T _A_CAM-A0 -1 T _A_CAM-F0

[0125] T _CAM-A0_CAM-G0 =T _A_CAM-A0 -1 T _A_CAM-G0

[0126] T _CAM-A0_CAMHB0 =T _A_CAM-A0 -1 T _A_CAM-H0

[0127] T _CAM-A0_CAM-B0 =T _A_CAM-A0 -1 T _A_CAM-B0

[0128] in,

[0129] T _CAM-A0_CAM-A0 is the external parameter of the calibration camera CAM-A0 relative to the calibration camera CAM-A0;

[0130] T _CAM-A0_CAM-B0 It is the external parameter of the calibration camera CAM-B0 relative to the calibration camera CAM-A0;

[0131] T _CAM-A0_CAM-C0 It is the external parameter of the calibration camera CAM-C0 relative to the calibration camera CAM-A0;

[0132] T _CAM-A0_CAM-D0 It is the external parameter of the calibration camera CAM-D0 relative to the calibration camera CAM-A0;

[0133] T_CAM-A0_CAM-E0 It is the external parameter of the calibration camera CAM-E0 relative to the calibration camera CAM-A0;

[0134] T _CAM-A0_CAM-F0 It is the external parameter of the calibration camera CAM-F0 relative to the calibration camera CAM-A0;

[0135] T _CAM-A0_CAM-G0 It is the external parameter of the calibration camera CAM-G0 relative to the calibration camera CAM-A0;

[0136] T _CAM-A0_CAM-H0 It is the external parameter of the calibration camera CAM-H0 relative to the calibration camera CAM-A0.

[0137] Further, step S102 includes:

[0138] Step S102.1: Place a multi-target positioning device on a vehicle (e.g., install a calibrated eight-target positioning device on the roof of the vehicle to be calibrated). Each camera to be calibrated on the vehicle forms a camera group with several calibrated cameras on the multi-target positioning device. The cameras to be calibrated and the calibrated cameras in each camera group can observe the same chessboard grid together.

[0139] Step S102.2: performing pnp projection and epipolar constraint optimization on the camera to be calibrated and the calibration camera in each camera group to obtain relative extrinsic parameters between the cameras in each camera group;

[0140] Step S102.3: Based on the extrinsic parameters between the cameras in each camera group and the relative extrinsic parameters between each calibration camera, obtain the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-object calibration device.

[0141] like Figure 4 As shown, in this embodiment, there are 8 cameras to be calibrated on the vehicle, specifically: CAM-A1, CAM-A2, CAM-B1, CAM-B2, CAM-C1, CAM-C2, CAM-D1, and CAM-D2.

[0142] Therefore: CAM-A is a camera group, including: CAM-A0, CAM-A1 and CAM-A2. Similarly, there are camera groups CAM-B, CAM-C and CAM-D.

[0143] During the calibration process, the camera groups will be jointly calibrated first. For example, camera group CAM-A will be jointly calibrated first, followed by camera group CAM-B, then camera group CAM-C, and then camera group CAM-D.

[0144] Take the camera group CAM-A as an example. Joint calibration is performed on the camera group CAM-A, that is, the CAM-A1, CAM-A2 and CAM-A0 cameras in the CAM-A camera group observe the same chessboard together, establish the reference coordinate system on the chessboard, extract the chessboard corner points, perform pnp projection and polar constraint optimization on the CAM-A1, CAM-A2 and CAM-A0 cameras, generate the relative poses between CAM-A1-CAM-A2, CAM-A1-CAM-A0 and CAM-A2-CAM-A0, and establish local pose graphs for these relative poses.

[0145] Similarly, corresponding operations are performed on the CAM-B1-CAM-B2, CAM-B1-CAM-B0 and CAM-B2-CAM-B0 poses, the CAM-C1-CAM-C2, CAM-C1-CAM-C0 and CAM-C2-CAM-C0 poses, and the CAM-D1-CAM-D2, CAM-D1-CAM-D0 and CAM-D2-CAM-D0 poses.

[0146] Specifically,

[0147] The reference coordinate system B is established on the upper left corner of the chessboard. Using the chessboard plane assumption, the three-dimensional coordinate set P of n chessboard corner points in the reference coordinate system B is generated:

[0148] P=[p0 p1...p i ...p n-1 ]

[0149] n=width*height

[0150] Among them, the three-dimensional coordinates of the i-th corner point

[0151] p i =[X i Y i Z i ]

[0152] X i =(i%width)*squareSize

[0153] Y i =(i / width)*squareSize

[0154] Z i =0

[0155] Among them, width is the width of the chessboard, height is the height of the chessboard, and squareSize is the length of the chessboard.

[0156] Assume that the position of camera CAM-A0 in reference coordinate system B is T _B_CAM-A0 , assuming that the position of camera CAM-A1 in reference coordinate system B is T _B_CAM-A1 , assuming that the position of camera CAM-A2 in reference coordinate system B is T _B_CAM-A2 .

[0157] For camera CAM-A0:

[0158]

[0159] in, Indicates p i Three-dimensional coordinates in the CAM-A0 coordinate system.

[0160] Will Normalization:

[0161]

[0162] Define the pnp residual function of CAM-A0

[0163]

[0164] in,

[0165] K CAM-A0 is the internal reference of CAM-A0;

[0166] is the pixel coordinate of the i-th corner point observed by CAM-A0.

[0167] Minimize the pnp residual function That is, T _B_CAM-A0 .

[0168] Similarly, the pnp residual function can be obtained It can be obtained that T _B_CAM-A1 , T _B_CAM-A2 .

[0169] For cameras CAM-A0 and CAM-A1, calculate the fundamental matrix F 01 :

[0170]

[0171] in,

[0172] K CAM-A0 is the internal reference of CAM-A0;

[0173] K CAM-A1 It is the internal reference of CAM-A1;

[0174] R_CAM-A0_CAM-A1 is the rotation of CAM-A1 relative to CAM-A0;

[0175] t _CAM-A0_CAM-A1 is the translation of CAM-A1 relative to CAM-A0;

[0176] t _CAM-A0_CAM-A1 The antisymmetric matrix of .

[0177] Define the epipolar constraint residual function of camera CAM-A0 and camera CAM-A1

[0178]

[0179]

[0180] in,

[0181] and is the pixel coordinate of the i-th corner point observed by CAM-A0;

[0182] and is the pixel coordinate of the i-th corner point observed by CAM-A1.

[0183] Minimize the epipolar constraint residual function That is, we get t _CAM-A0_CAM-A1 and R _CAM-A0_CAM-A1 .

[0184] Similarly, we can get the polar constraint residual function and It can be obtained that t _CAM-A0_CAM-A2 , R _CAM-A0_CAM-A2 and t _CAM-A1_CAM-A2 , R _CAM-A1_CAM-A2 .

[0185] Define the joint optimization overall residual function r opt :

[0186]

[0187] Minimize the overall residual function of the joint optimization, that is, get T _B_CAM-A0 , T _B_CAM-A1 , T _B_CAM-A2 .

[0188] Get the relative pose T between CAM-A1-CAM-A0 and CAM-A2-CAM-A0 _CAM-A0_CAM-A1 , T _CAM-A0_CAM-A2 :

[0189] T_CAM-A0_CAM-A1 =T _B_CAM-A0 -1 T _B_CAM-A1

[0190] T _CAM-A0_CAM-A2 =T _B_CAM-A0 -1 T _B_CAM-A2

[0191] Similarly, the external parameter T of camera group CAM-B can be obtained _CAM-B0_CAM-B1 , T _CAM-B0_CAM-B2 .

[0192] Similarly, the external parameter T of the camera group CAM-C can be obtained _CAM-C0_CAM-C1 , T _CAM-C0_CAM-C2 .

[0193] Similarly, the external parameter T of the camera group CAM-D can be obtained _CAM-D0_CAM-D1 , T _CAM-D0_CAM-D2 .

[0194] Because the relative external parameters between the calibrated cameras in the multi-target calibration device have been obtained through step S101, all camera groups CAM-A, CAM-B, CAM-C and CAM-D are jointly calibrated, and the accurate relative poses between the camera groups can be obtained.

[0195] Transform the camera pose to the coordinate system of camera CAM-A0, and we get:

[0196] T _CAM-A0_CAM-A1 =T _CAM-A0_CAM-A0 T _CAM-A0_CAM-A1

[0197] T _CAM-A0_CAM-A2 =T _CAM-A0_CAM-A0 T _CAM-A0_CAM-A2

[0198] T _CAM-A0_CAM-B1 =T _CAM-A0_CAM-B0 T _CAM-B0_CAM-B1

[0199] T _CAM-A0_CAM-B2 =T _CAM-A0_CAM-B0 T _CAM-B0_CAM-B2

[0200] T _CAM-A0_CAM-C1 =T _CAM-A0_CAM-C0 T _CAM-C0_CAM-C1

[0201] T _CAM-A0_CAM-C2 =T _CAM-A0_CAM-C0 T _CAM-C0_CAM-C2

[0202] T _CAM-A0_CAM-D1 =T _CAM-A0_CAM-D0 T _CAM-D0_CAM-D1

[0203] T _CAM-A0_CAM-D2 =T _CAM-A0_CAM-D0 T _CAM-D0_CAM-D2

[0204] Further, step S103 includes:

[0205] Step S103.1: Obtain the translation measurement value of each camera to be calibrated on the vehicle in the vehicle body coordinate system baselink;

[0206] Step S103.2: Based on the translation measurement value of each camera to be calibrated in the vehicle body coordinate system baselink, the factor graph optimization method is used to optimize the external parameters of any calibration camera in the multi-target calibration device in the vehicle body coordinate system baselink;

[0207] Step S103.3: Based on the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device and the extrinsic parameters of the optimized calibration camera in the vehicle body coordinate system baselink, obtain the extrinsic parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink.

[0208] The construction of factor graph is as follows Figure 5 The orange edges represent the external parameters obtained in step S101, the gray edges represent the external parameters obtained in step S102, and the blue edges represent the translation measurement values ​​from the camera group to the vehicle body coordinate system baselink. Finally, the factor graph is optimized to obtain the external parameters of the vehicle camera group relative to the vehicle body coordinate system baselink.

[0209] R _baselink_CAM-A0 ,t _baselink_CAM-A0 To calibrate the rotation and translation of the camera CAM-A0 in the vehicle body coordinate system baselink, it is used as the optimization variable of the factor graph optimization.

[0210] t _baselink_CAM-A1 is the translation measurement value of the camera CAM-A1 to be calibrated in the vehicle body coordinate system baselink.

[0211] t _baselink_CAM-A2 is the translation measurement value of the camera CAM-A2 to be calibrated in the vehicle body coordinate system baselink.

[0212] t _baselink_CAM-B1 is the translation measurement value of the camera CAM-B1 to be calibrated in the vehicle body coordinate system baselink.

[0213] t _baselink_CAM-B2 is the translation measurement value of the camera CAM-B2 to be calibrated in the vehicle body coordinate system baselink.

[0214] t _baselink_CAM-C1is the translation measurement value of the camera CAM-C1 to be calibrated in the body coordinate system baselink.

[0215] t _baselink_CAM-C2 is the translation measurement value of the camera CAM-C2 to be calibrated in the body coordinate system baselink.

[0216] t _baselink_CAM-D1 is the translation measurement value of the camera CAM-D1 to be calibrated in the vehicle body coordinate system baselink.

[0217] t _baselink_CAM-D2 It is the translation measurement value of the camera CAM-D2 to be calibrated in the body coordinate system baselink.

[0218] Define the factor graph to optimize the residual function r baselink :

[0219] r baselink =r _baselink_CAM-A1 +r _baselink_CAM-A2 +r _baselink_CAM-B1

[0220] +r _baselink_CAM-B2 +r _baselink_CAM-C1 +r _baselink_CAM-C2

[0221] +r _baselink_CAM-D1 +r _baselink_CAM-D2

[0222] r _baselink_CAM-A1

[0223] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-A1 +t _baselink_CAM-A0 )

[0224] -t _baselink_CAM-A1 || 2

[0225] r _baselink_CAM-A2

[0226] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-A2 +t _baselink_CAM-A0 )

[0227] -t _baselink_CAM-A2 || 2

[0228] r _baselink_CAM-B1

[0229] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-B1 +t _baselink_CAM-A0 )

[0230] -t_baselink_CAM-B1 || 2

[0231] r _baselink_CAM-B2

[0232] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-B2 +t _baselink_CAM-A0 )

[0233] -t _baselink_CAM-B2 || 2

[0234] r _baselink_CAM-C1

[0235] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-C1 +t _baselink_CAM-A0 )

[0236] -t _baselink_CAM-C1 || 2

[0237] r _baselink_CAM-C2

[0238] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-C2 +t _baselink_CAM-A0 )-t _baselink_CAM-C2 || 2

[0239] r _baselink_CAM-D1

[0240] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-D1 +t _baselink_CAM-A0 )-t _baselink_CAM-D1 || 2

[0241] r _baselink_CAM-D2

[0242] =||(R _baselink_CAM-A0 t _CAM-A0_CAM-D2 +t _baselink_CAM-A0 )-t _baselink_CAM-D2 || 2

[0243] Minimize the above factor graph to optimize the residual function r baselink , that is, R _baselink_CAM-A0 ,t _baselink_CAM-A0 .

[0244] Calibrate the position and posture of the camera CAM-A0 in the vehicle body coordinate system baselink T _baselink_CAM-A0 :

[0245]

[0246] Transform the camera pose to the baselink coordinate system of the vehicle body coordinate system, and we get:

[0247] T _baselink_CAM-A1 =T _baselink_CAM-A0 T _CAM-A0_CAM-A1

[0248] T _baselink_CAM-A2 =T _baselink_CAM-A0 T _CAM-A0_CAM-A2

[0249] T _baselink_CAM-B1 =T _baselink_CAM-A0 T _CAM-A0_CAM-B1

[0250] T _baselink_CAM-B2 =T _baselink_CAM-A0 T _CAM-A0_CAM-B2

[0251] T _baselink_CAM-C1 =T _baselink_CAM-A0 T _CAM-A0_CAM-C1

[0252] T _baselink_CAM-C2 =T _baselink_CAM-A0 T _CAM-A0_CAM-C2

[0253] T _baselink_CAM-D1 =T _baselink_CAM-A0 T _CAM-A0_CAM-D1

[0254] T _baselink_CAM-D2 =T _baselink_CAM-A0 T _CAM-A0_CAM-D2

[0255] T _baselink_CAM-A1 , T _baselink_CAM-A2 , T _baselink_CAM-B1 , T _baselink_CAM-B2 , T _baselink_CAM-C1 , T _baselink_CAM-C2 , T _baselink_CAM-D1 , T _baselink_CAM-D2 That is, the external parameter of the vehicle camera group relative to the vehicle body coordinate system baselink.

[0256] It should be noted that, in the embodiments of the present invention, relative posture and relative external parameters have the same meaning.

[0257] In summary, through the method of the present invention, the external parameters of the camera to be calibrated on the vehicle relative to the vehicle body coordinate system can be obtained, and the translation accuracy of the external parameters can reach 1-2mm, and the angle accuracy can reach 0.05-0.15 degrees.

[0258] The schematic diagram of the result after stereo calibration using the external parameters of the binocular camera calibrated by the present invention is as follows: Figure 6As shown in the figure, even objects far away can be aligned, which shows that the calibration accuracy of the present invention is very high. The camera extrinsic parameters calibrated by the present invention, the projection results of the panoramic camera and the laser radar are as follows: Figure 7 As shown, the surround camera and lidar projection results are as follows Figure 8 shown.

[0259] In some other embodiments, the present invention discloses a camera calibration device, which uses a multi-target calibration device equipped with multiple calibration cameras to implement extrinsic parameter calibration of a camera to be calibrated of a vehicle.

[0260] like Fig. 9 As shown, the camera calibration device includes:

[0261] The hand-eye calibration module 201 is used to obtain the relative external parameters between each calibration camera on the multi-target calibration device based on the running trajectory method;

[0262] A joint calibration module 202 is used to place a multi-object calibration device on a vehicle, and the camera to be calibrated on the vehicle and the calibration camera on the multi-object calibration device have a common viewing area;

[0263] Based on the relative extrinsic parameters between each calibrated camera, the relative extrinsic parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device are obtained by using pnp projection and epipolar constraint optimization.

[0264] The factor graph optimization module 203 is used to obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink by using a factor graph optimization method based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device.

[0265] Furthermore, the hand-eye calibration module includes:

[0266] A motion unit, a device body for a multi-target calibration device, carries multiple calibration cameras for movement, and at each point of the movement, each calibration camera is aligned with the chessboard in turn to capture images;

[0267] A first running trajectory determining unit, used for obtaining the running trajectory of each calibration camera of the multi-target calibration device according to the image taken by each calibration camera;

[0268] A second running track determination unit is used to obtain the running track of the device body of the multi-target positioning device according to the motion point position of the multi-target positioning device;

[0269] A first external parameter determination unit, configured to obtain an external parameter of each calibration camera relative to the device body based on the running trajectory of the device body and the running trajectory of each calibration camera;

[0270] The second extrinsic parameter determination unit obtains relative extrinsic parameters between each calibration camera based on the extrinsic parameters of each calibration camera relative to the device body.

[0271] Furthermore, the joint calibration module includes:

[0272] A placement unit is used to place the multi-object calibration device on the vehicle, and each camera to be calibrated on the vehicle forms a camera group with a plurality of calibration cameras on the multi-object calibration device, and the camera to be calibrated and the calibration camera in each camera group can observe the same chessboard grid together;

[0273] A third extrinsic parameter determination unit is used to perform pnp projection and epipolar constraint optimization on the camera to be calibrated and the calibration camera in each camera group to obtain relative extrinsic parameters between the cameras in each camera group;

[0274] The fourth extrinsic parameter determination unit is used to obtain the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device based on the extrinsic parameters between each camera in each camera group and the relative extrinsic parameters between each calibration camera.

[0275] Furthermore, the factor graph optimization module includes:

[0276] A translation measurement value acquisition unit, used to acquire the translation measurement value of each camera to be calibrated on the vehicle in the vehicle body coordinate system baselink;

[0277] A factor graph optimization unit is used to optimize the external parameters of any calibration camera in the multi-target calibration device in the body coordinate system baselink based on the translation measurement value of each camera to be calibrated in the body coordinate system baselink by using a factor graph optimization method;

[0278] The fifth extrinsic parameter determination unit is used to obtain the extrinsic parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink based on the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device and the extrinsic parameters of the optimized calibration camera in the vehicle body coordinate system baselink.

[0279] Furthermore, it should be noted that: when the camera calibration device provided in the above embodiment determines the dominant vertex, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the camera calibration device is divided into different functional modules to complete all or part of the functions described above.

[0280] In addition, the camera calibration device and the camera calibration method provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here.

[0281] In addition, in some other embodiments, Fig.10 As shown, the present invention also discloses a computing device, including:

[0282] One or more processors 301;

[0283] Memory 302;

[0284] and one or more programs, wherein the one or more programs are stored in the memory 302 and configured to be executed by the one or more processors 301, and the one or more programs include instructions of the camera calibration method disclosed in the above embodiment.

[0285] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0286] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the camera calibration method provided by the method embodiment of the present invention.

[0287] In addition, the computing device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 301, the memory 302 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0288] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0289] In addition, in some other embodiments, the present invention further discloses a storage medium, wherein the storage medium stores one or more computer-readable programs, wherein the one or more programs include instructions, and the instructions are suitable for being loaded by a memory and executing the camera calibration method disclosed in the above embodiment.

[0290] The present invention discloses a camera calibration method, device, equipment and storage medium, which have the following

[0291] Beneficial effects:

[0292] First, the present invention discloses a multi-target positioning device, which has a simple structure and can obtain the relative external parameters between each calibration camera on the multi-target positioning device by analyzing the running trajectory. The multi-target positioning device can be applied to camera calibration occasions.

[0293] Second, the present invention uses a multi-target calibration device to calibrate the camera on the vehicle, and adopts pnp projection and epipolar constraint optimization to obtain the relative external parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device.

[0294] Third, the present invention adopts a factor graph optimization method to obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink.

[0295] Fourth, the present invention can achieve high-precision, fast, outdoor-calibration, and highly generalized multi-camera joint calibration.

[0296] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A camera calibration method, characterized in that: Using a multi-target calibration device equipped with multiple calibration cameras to calibrate the external parameters of the camera to be calibrated of the vehicle; The camera calibration method comprises: Step S1: Based on the running trajectory method, the relative external parameters between each calibration camera on the multi-target calibration device are obtained; Step S2: placing a multi-object calibration device on a vehicle, and the camera to be calibrated on the vehicle and the calibration camera on the multi-object calibration device have a common viewing area; Based on the relative extrinsic parameters between each calibrated camera, the relative extrinsic parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device are obtained by using pnp projection and epipolar constraint optimization. Step S3: Based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device, the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink are obtained by using a factor graph optimization method.

2. The camera calibration method according to claim 1, characterized in that: The step S1 comprises: Step S1.1: The main body of the multi-target calibration device carries multiple calibration cameras and moves, and at each point of the movement, each calibration camera is aligned with the chessboard in turn to capture images; Step S1.2: obtaining the running trajectory of each calibration camera of the multi-target calibration device according to the images taken by each calibration camera; Step S1.3: obtaining the running track of the device body of the multi-target positioning device according to the moving points of the multi-target positioning device; Step S1.4: based on the running trajectory of the device body and the running trajectory of each calibration camera, obtaining the external parameters of each calibration camera relative to the device body; Step S1.5: Based on the external parameters of each calibration camera relative to the device body, obtain the relative external parameters between each calibration camera.

3. The camera calibration method according to claim 1, characterized in that: The step S2 comprises: Step S2.1: a multi-object calibration device is placed on a vehicle, and each camera to be calibrated on the vehicle forms a camera group with several calibration cameras on the multi-object calibration device, and the cameras to be calibrated and the calibration cameras in each camera group can observe the same chessboard grid together; Step S2.2: performing pnp projection and epipolar constraint optimization on the camera to be calibrated and the calibration camera in each camera group to obtain relative external parameters between the cameras in each camera group; Step S2.3: Based on the extrinsic parameters between the cameras in each camera group and the relative extrinsic parameters between each calibration camera, the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-object calibration device are obtained.

4. The camera calibration method according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Obtain the translation measurement value of each camera to be calibrated on the vehicle in the vehicle body coordinate system baselink; Step S3.2: Based on the translation measurement value of each camera to be calibrated in the vehicle body coordinate system baselink, the factor graph optimization method is used to optimize the external parameters of any calibration camera in the multi-target calibration device in the vehicle body coordinate system baselink; Step S3.3: Based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device and the external parameters of the optimized calibration camera in the vehicle body coordinate system baselink, obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink.

5. A camera calibration device, characterized in that: Using a multi-target calibration device equipped with multiple calibration cameras to calibrate the external parameters of the camera to be calibrated of the vehicle; The camera calibration device comprises: The hand-eye calibration module is used to obtain the relative external parameters between each calibration camera on the multi-target calibration device based on the running trajectory method; A joint calibration module is used to place a multi-object calibration device on a vehicle, and the camera to be calibrated on the vehicle and the calibration camera on the multi-object calibration device have a common viewing area; Based on the relative extrinsic parameters between each calibrated camera, the relative extrinsic parameters between each camera to be calibrated on the vehicle and any calibrated camera in the multi-target calibration device are obtained by using pnp projection and epipolar constraint optimization. The factor graph optimization module is used to obtain the external parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink by using the factor graph optimization method based on the relative external parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device.

6. The camera calibration device according to claim 5, characterized in that: The hand-eye calibration module comprises: A motion unit, a device body for a multi-target calibration device, carries multiple calibration cameras for movement, and at each point of the movement, each calibration camera is aligned with the chessboard in turn to capture images; A first running trajectory determining unit, used for obtaining the running trajectory of each calibration camera of the multi-target calibration device according to the image taken by each calibration camera; A second running track determination unit is used to obtain the running track of the device body of the multi-target positioning device according to the motion point position of the multi-target positioning device; A first external parameter determination unit, configured to obtain an external parameter of each calibration camera relative to the device body based on the running trajectory of the device body and the running trajectory of each calibration camera; The second extrinsic parameter determination unit obtains relative extrinsic parameters between each calibration camera based on the extrinsic parameters of each calibration camera relative to the device body.

7. The camera calibration device according to claim 5, characterized in that: The joint calibration module comprises: A placement unit is used to place the multi-object calibration device on the vehicle, and each camera to be calibrated on the vehicle forms a camera group with a plurality of calibration cameras on the multi-object calibration device, and the camera to be calibrated and the calibration camera in each camera group can observe the same chessboard grid together; A third extrinsic parameter determination unit is used to perform pnp projection and epipolar constraint optimization on the camera to be calibrated and the calibration camera in each camera group to obtain relative extrinsic parameters between the cameras in each camera group; The fourth extrinsic parameter determination unit is used to obtain the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device based on the extrinsic parameters between each camera in each camera group and the relative extrinsic parameters between each calibration camera.

8. The camera calibration device according to claim 5, characterized in that: The factor graph optimization module includes: A translation measurement value acquisition unit, used to acquire the translation measurement value of each camera to be calibrated on the vehicle in the vehicle body coordinate system baselink; A factor graph optimization unit is used to optimize the external parameters of any calibration camera in the multi-target calibration device in the body coordinate system baselink based on the translation measurement value of each camera to be calibrated in the body coordinate system baselink by using a factor graph optimization method; The fifth extrinsic parameter determination unit is used to obtain the extrinsic parameters of each camera to be calibrated relative to the vehicle body coordinate system baselink based on the relative extrinsic parameters between each camera to be calibrated and any calibration camera in the multi-target calibration device and the extrinsic parameters of the optimized calibration camera in the vehicle body coordinate system baselink.

9. A computing device, characterized in that include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions of the camera calibration method described in any one of claims 1 to 4.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by the memory and executing the camera calibration method described in any one of claims 1 to 4.

Images