A Multi-View Camera Calibration Method and System Based on Nonlinear Optimization

Abstract

The present invention relates to a multi-view camera calibration method and system based on nonlinear optimization. The method comprises: pre-calibrating the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated; roughly calibrating the extrinsic parameters of all cameras to be calibrated in a selected world coordinate system through the adjacent relationship between multiple cameras to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated; and synchronously optimizing the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix through a nonlinear optimization method to obtain refined camera extrinsic parameters. Aiming at the extrinsic parameter calibration problem of multi-view cameras, the present invention adopts a method of rough calibration first and then optimization, which effectively solves the self-occlusion problem of two-dimensional checkerboard calibration objects in a multi-camera system. In the case of not needing a three-dimensional calibration object, the topological relationship between multiple cameras is used to synchronously optimize all camera extrinsic parameters, so as to obtain a calibration result with considerable accuracy and improve the calibration accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-view camera calibration method and system based on nonlinear optimization. Background Art

[0002] Camera calibration is a key technology involved in many research topics in computer vision and computer graphics. Whether in image measurement or machine vision applications, the calibration of camera parameters is a very critical link. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the camera. Therefore, doing a good job of camera calibration is the prerequisite for doing a good job in subsequent work, and improving the calibration accuracy is the focus of scientific research. Camera calibration methods are roughly divided into traditional camera calibration methods, camera self-calibration methods, and active vision camera calibration methods. The traditional camera calibration method refers to the use of some calibration objects with obvious corner points and known sizes for calibration. According to the different calibration objects, it can be divided into two-dimensional plane calibration objects and three-dimensional calibration objects. The two-dimensional calibration object is simple to make and has considerable accuracy, but it needs to take multiple sets of images during calibration. The three-dimensional calibration object can be calibrated by a single image, with high calibration accuracy, but the production cost is relatively high. The self-calibration method refers to the calibration of the camera using camera motion constraints or scene constraints. This calibration method does not require calibration objects and can be calibrated online, but the disadvantage is that the algorithm robustness is poor. The camera calibration method of active vision refers to calibrating the camera when certain motion information of the camera is known. This method does not require calibration objects, has a simple algorithm and high robustness, but it needs to control the camera to perform certain specific movements and the experimental requirements are relatively strict.

[0003] In the pinhole camera model, the imaging process of an object in a world coordinate system includes four coordinate system transformation processes - from the world coordinate system to the camera coordinate system, from the camera coordinate system to the image plane coordinate system, and from the image plane coordinate system to the pixel plane coordinate system. The first transformation process involves the camera extrinsic parameters, while the second two transformations use the camera intrinsic parameters. Therefore, calibrating the camera's intrinsic and extrinsic parameters is particularly important in the process of camera imaging and reprojection. Dr. Laura Ribeiro proposed a calibration method for small baseline multi-view cameras, which uses a two-dimensional chessboard plane as a calibration object. This method overcomes the shortcomings of the high-precision calibration object required by the traditional calibration method, and only a printed chessboard is needed; at the same time, compared with self-calibration, the accuracy is improved; compared with the active vision camera calibration method, it is easier to operate. However, this method is particularly effective when there is no self-occlusion of the calibration object in the camera shooting angle, but the calibration accuracy is poor when there is self-occlusion of the calibration object in the camera shooting angle. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-view camera calibration method and system based on nonlinear optimization, so as to effectively solve the self-occlusion problem of two-dimensional checkerboard calibration objects in a multi-camera system, and obtain a calibration result with considerable accuracy without the need for three-dimensional calibration objects.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A multi-view camera calibration method based on nonlinear optimization, comprising:

[0007] Pre-calibrate the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated;

[0008] The external parameters of all cameras to be calibrated in the selected world coordinate system are roughly calibrated through the neighboring relationship between multiple cameras, and the initial external parameters to be optimized of all cameras to be calibrated are obtained;

[0009] By using a nonlinear optimization method, the initial extrinsic parameters to be optimized of all cameras to be calibrated are synchronously optimized based on the intrinsic parameter matrix to obtain refined camera extrinsic parameters.

[0010] Optionally, the pre-calibrating the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrices of all cameras to be calibrated specifically includes:

[0011] The CameraCalibration calibration toolbox built into the Matlab software is used to pre-calibrate the internal parameters of each camera to be calibrated, and the internal parameter matrix of all cameras to be calibrated is obtained.

[0012] Optionally, the step of roughly calibrating the extrinsic parameters of all cameras to be calibrated in a selected world coordinate system by using the neighbor relationship between multiple cameras to obtain initial extrinsic parameters to be optimized of all cameras to be calibrated specifically includes:

[0013] Selecting a camera coordinate system of any camera among all the cameras to be calibrated as the world coordinate system of the entire camera system, and calculating the coordinates of all the cameras to be calibrated in the world coordinate system;

[0014] Determine the pairwise adjacent relationship between all the cameras to be calibrated according to the coordinates of all the cameras to be calibrated in the world coordinate system;

[0015] Calibrate the external parameters of all the cameras to be calibrated according to the pairwise neighbor relationship between all the cameras to be calibrated;

[0016] The extrinsic parameters of all cameras to be calibrated are uniformly transferred to the world coordinate system by utilizing the pairwise adjacent relationship between all cameras to be calibrated, so as to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated in the world coordinate system.

[0017] Optionally, the step of synchronously optimizing the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix by a nonlinear optimization method to obtain refined camera extrinsic parameters specifically includes:

[0018] Obtain chessboard images of different postures taken synchronously by all cameras to be calibrated;

[0019] Detecting corner points on the chessboard images in different postures to obtain corner points in all postures of the chessboard;

[0020] For each chessboard posture, reproject the corner points under the chessboard posture back to the corresponding chessboard image taken by each camera to be calibrated according to the intrinsic parameter matrix and the initial extrinsic parameter to obtain the reprojection points;

[0021] Calculating the Euclidean distance between the reprojection point and the corresponding corner point;

[0022] The Euclidean distance is minimized by the LM optimization algorithm to obtain refined camera extrinsics.

[0023] Optionally, the method further comprises:

[0024] The reprojection verification method is used to quantitatively measure the accuracy of the refined camera extrinsic calibration results.

[0025] A multi-view camera calibration system based on nonlinear optimization, comprising:

[0026] An intrinsic parameter pre-calibration module is used to pre-calibrate the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrices of all cameras to be calibrated;

[0027] The external parameter rough calibration module is used to roughly calibrate the external parameters of all cameras to be calibrated in the selected world coordinate system through the neighboring relationship between multiple cameras, and obtain the initial external parameters to be optimized of all cameras to be calibrated;

[0028] The extrinsic parameter optimization module is used to synchronously optimize the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix through a nonlinear optimization method to obtain refined camera extrinsic parameters.

[0029] Optionally, the internal reference pre-calibration module specifically includes:

[0030] The intrinsic parameter pre-calibration unit is used to pre-calibrate the internal parameters of each camera to be calibrated separately using the CameraCalibration calibration toolbox built into the Matlab software to obtain the intrinsic parameter matrix of all cameras to be calibrated.

[0031] Optionally, the external parameter rough calibration module specifically includes:

[0032] A camera coordinate calculation unit, used for selecting a camera coordinate system of any camera among all the cameras to be calibrated as a world coordinate system of the entire camera system, and calculating the coordinates of all the cameras to be calibrated in the world coordinate system;

[0033] An adjacent relationship determining unit, used to determine the adjacent relationship between all the cameras to be calibrated according to the coordinates of all the cameras to be calibrated in the world coordinate system;

[0034] An external parameter rough calibration unit, used for calibrating the external parameters of every two cameras among all the cameras to be calibrated according to the pairwise neighbor relationship between all the cameras to be calibrated;

[0035] The initial extrinsic parameter calculation unit is used to uniformly transfer the extrinsic parameters of all cameras to be calibrated to the world coordinate system by utilizing the pairwise adjacent relationship between all cameras to be calibrated, so as to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated in the world coordinate system.

[0036] Optionally, the external parameter optimization module includes:

[0037] A calibration image acquisition unit, used to acquire chessboard images of different postures synchronously taken by all cameras to be calibrated;

[0038] A corner point detection unit, used to detect corner points on the chessboard images with different postures, and obtain corner points in all postures of the chessboard;

[0039] A reprojection unit, for reprojecting, for each chessboard posture, the corner points under the chessboard posture back to the corresponding chessboard image taken by each camera to be calibrated according to the intrinsic parameter matrix and the initial extrinsic parameter, to obtain a reprojection point;

[0040] A Euclidean distance calculation unit, used to calculate the Euclidean distance between the reprojection point and the corresponding corner point;

[0041] The extrinsic parameter optimization unit is used to minimize the Euclidean distance through the LM optimization algorithm to obtain refined camera extrinsic parameters.

[0042] Optionally, the system further comprises:

[0043] The calibration result verification module is used to quantitatively measure the accuracy of the refined camera extrinsic calibration result by using a reprojection verification method.

[0044] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0045] The present invention provides a multi-view camera calibration method and system based on nonlinear optimization, the method comprising: pre-calibrating the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated; roughly calibrating the extrinsic parameters of all cameras to be calibrated in a selected world coordinate system through the adjacent relationship between multiple cameras to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated; and synchronously optimizing the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix through a nonlinear optimization method to obtain refined camera extrinsic parameters. Aiming at the extrinsic parameter calibration problem of multi-view cameras, the present invention adopts a method of rough calibration first and then optimization, which effectively solves the self-occlusion problem of two-dimensional checkerboard calibration objects in a multi-camera system. In the case of not needing a three-dimensional calibration object, the topological relationship between multiple cameras is used to synchronously optimize all camera extrinsic parameters, so as to obtain a calibration result with considerable accuracy and improve the calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 This is a flow chart of a multi-view camera calibration method based on nonlinear optimization according to the present invention;

[0048] Figure 2 A schematic diagram of a multi-view camera calibration method based on nonlinear optimization according to the present invention;

[0049] Figure 3 A schematic diagram of a multi-camera system provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a collection method for optimizing a data set provided in an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of verifying the accuracy of calibration results by reprojecting actual data set corner points based on experimental calibration of real camera extrinsics provided in an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of verifying the accuracy of calibration results by reprojecting virtual data set corner points based on experimental calibration of virtual camera extrinsics provided in an embodiment of the present invention; wherein the perspective without a chessboard in the image is the back of the virtual chessboard being captured. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The purpose of the present invention is to provide a multi-view camera calibration method and system based on nonlinear optimization, so as to effectively solve the self-occlusion problem of two-dimensional checkerboard calibration objects in a multi-camera system, and obtain a calibration result with considerable accuracy without the need for three-dimensional calibration objects.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Figure 1 This is a flow chart of a multi-view camera calibration method based on nonlinear optimization in the present invention. Figure 2 Schematic diagram of the process of a multi-view camera calibration method based on nonlinear optimization in the present invention. Figure 1 and Figure 2 The present invention provides a multi-view camera calibration method based on nonlinear optimization, comprising:

[0057] Step 101: pre-calibrate the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrices of all cameras to be calibrated.

[0058] In this step, the intrinsic parameters of all cameras to be calibrated (hereinafter referred to as cameras) are pre-calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated and use it as a known quantity. Specifically, in order to obtain the internal parameters of all cameras for use in the subsequent reprojection stage, the built-in CameraCalibration toolbox of matlab is used to pre-calibrate the internal parameters of all cameras (referred to as intrinsic parameters) to obtain the intrinsic parameter matrix K of all cameras. i .

[0059] The purpose of the camera intrinsic parameter pre-calibration stage of the present invention is mainly to calibrate the internal parameters of the camera. The internal parameters mainly include camera distortion, focal length, principal point position, etc. The calibration result is the radial distortion parameters (k1, k2) of the camera and the internal parameter matrix Where (f x ,f y ) represents the normalized focal length of the camera in the direction of the two image axes, (c x ,c y ) represents the coordinates of the camera principal point. K iis the set of intrinsic parameter matrices of all cameras to be calibrated. The parameters obtained in this stage will be used as constants in the projection calculation process of the subsequent extrinsic parameter calibration stage.

[0060] Therefore, the step 101 pre-calibrates the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated, which specifically includes:

[0061] The CameraCalibration calibration toolbox built into the Matlab software is used to pre-calibrate the internal parameters of each camera to be calibrated, and the internal parameter matrix of all cameras to be calibrated is obtained.

[0062] Step 102: Roughly calibrate the extrinsic parameters of all cameras to be calibrated in the selected world coordinate system through the neighbor relationship between multiple cameras, and obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated.

[0063] In this step, the initial extrinsic parameters of all cameras in the selected world coordinate system are calibrated by the neighboring relationship of the cameras. This step obtains the relatively rough camera extrinsic parameters to be optimized. Specifically, the camera coordinate system of one camera is selected as the world coordinate system of the entire camera system. In order to obtain the coordinates of all cameras in this world coordinate system, the extrinsic parameters of the two cameras are first calibrated according to the neighboring relationship between the two cameras (R ij ,t ij ), where R ij is a 3*3 rotation matrix, t ij is a three-dimensional column vector. Then, the camera neighbor relationship is also used to uniformly transfer the external parameters of all cameras to the world coordinate system. This step obtains the initial external parameters of all cameras in the world coordinate system (R iraw ,t iraw ). Since there is a certain calibration error in the initial external parameter, this error will be transmitted step by step during the transmission process, and the final external parameter has a large error, so this external parameter needs to be optimized.

[0064] The purpose of the camera extrinsic coarse calibration stage of the present invention is to roughly calibrate the extrinsic parameters of all cameras in the world coordinate system to serve as the initial parameters of the nonlinear optimization process in the next stage. In this stage, for a multi-camera system where all cameras are arranged in a circle and focus on the same central area, the camera coordinate system of one of the cameras is selected as the world coordinate system. According to the adjacent relationship of the cameras, the extrinsic parameters (R ij ,t ij ), the calibration method is:

[0065] When camera 1 and camera 2 simultaneously shoot the same checkerboard calibration object, the external parameters (R p1 ,t p1 )、(R p2,t p2 ), the external parameter matrix of camera 2 relative to camera 1 is:

[0066]

[0067] Here it is assumed that two adjacent cameras can detect the checkerboard corners at the same time in at least one checkerboard posture.

[0068] For two non-adjacent cameras, there are two situations when the external parameters between them are required: one is that the two cameras can see the same chessboard at the same posture in their field of view at the same time. In this case, there is no need to use the intermediate camera as a medium to transmit the external parameters and the external parameters of the two cameras can be directly calculated; the other is that the two cameras cannot see the same chessboard at the same posture in their field of view at the same time. In this case, the transfer relationship between the cameras needs to be used. If camera 1 is adjacent to camera 2, and camera 2 is adjacent to camera 3, then the external parameters of camera 2 relative to camera 1 (R 12 ,t 12 ), the external parameter of camera 3 relative to camera 2 (R 23 ,t 23 ), then the external parameter matrix of camera 3 relative to camera 1 is:

[0069]

[0070] Using this transfer relationship, we can get the external parameters (R iw ,t iw ) as the initial external parameter (R iraw ,t iraw ).

[0071] Therefore, the step 102 roughly calibrates the external parameters of all cameras to be calibrated in the selected world coordinate system through the neighboring relationship between multiple cameras to obtain the initial external parameters to be optimized of all cameras to be calibrated, which specifically includes:

[0072] Selecting a camera coordinate system of any camera among all the cameras to be calibrated as the world coordinate system of the entire camera system, and calculating the coordinates of all the cameras to be calibrated in the world coordinate system;

[0073] Determine the pairwise adjacent relationship between all the cameras to be calibrated according to the coordinates of all the cameras to be calibrated in the world coordinate system;

[0074] Calibrate the external parameters of all the cameras to be calibrated according to the pairwise neighbor relationship between all the cameras to be calibrated;

[0075] The extrinsic parameters of all cameras to be calibrated are uniformly transferred to the world coordinate system by utilizing the pairwise adjacent relationship between all cameras to be calibrated, so as to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated in the world coordinate system.

[0076] Step 103: Using a nonlinear optimization method, based on the intrinsic parameter matrix, the initial extrinsic parameters to be optimized of all cameras to be calibrated are synchronously optimized to obtain refined camera extrinsic parameters.

[0077] In this step, all camera extrinsics are optimized synchronously based on the calibration data by a nonlinear optimization method to obtain more refined extrinsics. Specifically, the initial extrinsics (R iraw ,t iraw ), the camera external parameters are refined using nonlinear optimization methods. The specific method is as follows: for the calibration image taken by the i-th camera, according to its posture (R pi ,t pi ), calculate the attitude extrinsic parameters (R p ,t p ); For each chessboard calibration object in each posture, find the camera angle that can detect the corner point, and then convert the corner point (x dpi ,y dpi ) is back-projected onto the corresponding images of all other cameras to obtain the back-projected point (x rpj ,y rpj ); Calculate the back projection point (x rpj ,y rpj ) and the image detection corner point (x dpj ,y dpj ) Where P rpj represents the back-projection point of the j-th camera under the p-th calibration object pose, P rpj =(x rpj ,y rpj );P dpj represents the corner point of the jth camera under the pth calibration object pose, P dpj =(x dpj ,y dpj ), N represents the number of cameras, and M represents the number of calibration object postures. The Euclidean distance is minimized by the LM optimization algorithm, and the refined external parameters are finally obtained.

[0078] The purpose of the nonlinear optimization stage of the external parameters of the present invention is to optimize the initial external parameters (R iraw ,t iraw) is further refined. For a set of newly captured calibration images, detect the corner points on the image. If a checkerboard posture can detect corner points on some cameras, it can be projected into the world coordinate system according to the camera extrinsic parameter matrix. If a checkerboard posture can detect corner points on the image taken by the reference camera, the coordinates of the corner point in the world coordinate system can be used directly. The above method is used to obtain the world coordinates of the corner points in all postures of the checkerboard. Next, we only need to use the camera intrinsic parameters obtained in the camera intrinsic parameter pre-calibration stage and the camera extrinsic parameters obtained in the camera extrinsic parameter coarse calibration stage to reproject each checkerboard posture back to the corresponding image taken by each camera, so that the reprojected points can be obtained. Where K j represents the intrinsic parameter matrix of the jth camera; R j , t j They represent the external parameter rotation matrix and translation vector of the jth camera respectively; k j1 , k j2 represents the radial distortion coefficient of the jth camera, M pi Represents the three-dimensional coordinates of the i-th corner point in the p-th posture of the chessboard.

[0079] Assuming that the number of cameras is N, the number of chessboard poses is M, there are K feature points on the chessboard model plane, and the noise on the image follows an independent and identical distribution, the maximum likelihood estimate can be obtained by minimizing the following function:

[0080]

[0081] Where m jpi It represents the i-th detected corner point in the p-th posture of the chessboard captured by the j-th camera.

[0082] Therefore, the step 103 uses a nonlinear optimization method to synchronously optimize the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix to obtain refined camera extrinsic parameters, which specifically includes:

[0083] Obtain chessboard images of different postures taken synchronously by all cameras to be calibrated;

[0084] Detecting corner points on the chessboard images in different postures to obtain corner points in all postures of the chessboard;

[0085] For each chessboard posture, reproject the corner points under the chessboard posture back to the corresponding chessboard image taken by each camera to be calibrated according to the intrinsic parameter matrix and the initial extrinsic parameter to obtain the reprojection points;

[0086] Calculating the Euclidean distance between the reprojection point and the corresponding corner point;

[0087] The Euclidean distance is minimized by the LM optimization algorithm to obtain refined camera extrinsics.

[0088] In practical applications, the method further includes: using a reprojection verification method to quantitatively measure the accuracy of the refined camera extrinsic calibration result.

[0089] This step uses the reprojection verification method to quantitatively measure the accuracy of the calibration results. Specifically, for the optimized external parameters, the present invention proposes two methods to verify their accuracy:

[0090] The first method is to have all the cameras involved in the calibration re-shoot a set of verification data sets, re-project the corner points based on the calibrated external parameters, and quantitatively calculate the error between the re-projected corner points and the detected corner points. Through the experimental verification of the present invention, the re-projected points are basically consistent with the detected points, and the average error is within 10 pixels.

[0091] The second method is to develop a virtual camera simulation system in the form of software code. By manually setting the camera's internal and external parameters and shooting "virtual" calibration data in this simulation system, the extrinsic parameters are calibrated into the calibration algorithm. The accuracy of the calibration result can be quantitatively measured by comparing the error between the calibrated external parameters and the set external parameters. The first method can also be used to reproject and verify the corner error.

[0092] In the calibration method for internal and external parameters of multi-view cameras based on nonlinear optimization provided by the present invention, the camera model adopts a pinhole camera model, and the calibration object used is a chessboard. In order to calibrate the internal and external parameters of multiple industrial cameras in the same world coordinate system, the method of the present invention is as follows: for the calibration of camera internal parameters, the calibration toolbox built into the matlab software is directly used to calibrate each camera separately, and the calibration result is used as a known quantity; the calibration of camera external parameters is divided into two parts: preprocessing and external parameter optimization. The preprocessing part roughly calibrates the external parameters of all cameras in the world coordinate system through the adjacent relationship between multiple cameras. Due to the existence of transmission errors between cameras, the external parameters obtained in this step are usually inaccurate. Therefore, the main work of the optimization part is to refine the external parameters obtained in the preprocessing part through a nonlinear optimization method to reduce the influence caused by the transmission error between the camera external parameters. Therefore, the method of the present invention can effectively solve the self-occlusion problem of the two-dimensional chessboard calibration object in the multi-camera system, and can use the topological relationship between multiple cameras to synchronously optimize all camera external parameters without the need for a three-dimensional calibration object, so as to obtain a calibration result with considerable accuracy.

[0093] A specific embodiment is used below to illustrate the implementation process of the method of the present invention and verify the technical effect of the method of the present invention.

[0094] The experimental equipment of the embodiment of the present invention is eight industrial cameras of the same model, respectively referred to as camera 0 to camera 7. The eight cameras are arranged in a circle according to the number, and the focus is on the same central area, such as Figure 3 shown.

[0095] Step 1) Pre-calibration of internal reference

[0096] The internal parameters of the eight cameras were pre-calibrated. Specifically, each camera took about 10 groups of chessboard focus pictures with different postures, put them into the built-in toolbox Camera Calibrator of Matlab for calibration, and stored the calibration results as Matlab tables for use as constants.

[0097] Step 2) External parameter calibration data collection

[0098] This step requires two data collections. The first one is used for rough calibration of camera extrinsic parameters. Taking two adjacent cameras as units, more than 10 groups of chessboard images with different postures are taken within the line of sight of the two cameras. The second one is used for the extrinsic parameter optimization process. During the collection, all cameras need to synchronously shoot more than ten groups of chessboard corner points with different postures to ensure that each camera can detect the corner points in at least one group of postures. It is best to rotate the chessboard around the central axis at a uniform speed for one circle, and all cameras synchronously and continuously collect data, such as Figure 4 As shown, the acquired image data is named in the format of PositionXX_CameraXX.

[0099] Step 3) Rough calibration of camera extrinsics

[0100] The camera coordinate system of camera 0 is selected as the world coordinate system of the entire system. For the first collected data, the images with the same Position (chessboard posture) between each pair of cameras are used to calibrate the extrinsic parameters between the pair of cameras, including the three-dimensional rotation vector R in the form of Euler angles and the three-dimensional translation vector t. Then, the average extrinsic parameters of each group of extrinsic parameters are calculated as the extrinsic parameters of the two cameras. After obtaining the extrinsic parameters of each pair of adjacent cameras, the extrinsic parameters of all cameras in the world coordinate system are calculated using the camera adjacent relationship. The extrinsic parameters obtained in this step are used as the initial values ​​of the extrinsic parameters to be optimized.

[0101] Step 4) Corner reprojection

[0102] For the data collected for the second time, check whether the corner points can be detected in the picture taken by camera 0 and record them. If yes, save the corresponding corner point image coordinates; if not, sequentially check whether there are corner points in the images taken by the corresponding postures of cameras 1 to 7. If a corner point is detected, reproject the corner point to the image where camera 0 is located through the external parameters between the camera and camera 0 and the internal parameters of the two cameras, and stop detecting the remaining cameras. This step can ensure that there are corner points on all images of camera 0. Use the internal and external parameters of all cameras to reproject the corner points on the image of camera 0 to the corresponding image of each camera to obtain the reprojected points of all cameras.

[0103] Step 5) External parameter optimization

[0104] Obtain the reprojected points on all calibration images through step 4 Detect the corner points in all images taken by camera 1 to camera 7. If no corner points are detected, skip this image. If corner points are detected, calculate the following cost function:

[0105]

[0106] The Levenberg-Marquardt algorithm is used to iteratively minimize the cost function to find (R j , t j ) is the refined camera extrinsic parameter.

[0107] Step 6) Calibration result verification

[0108] After obtaining the optimized external parameters, the accuracy of the results needs to be verified. The present invention provides two verification methods: the first method is that all cameras involved in the calibration re-shoot a set of verification data sets, reproject the corner points according to the same logic as step 4) based on the calibrated external parameters, and quantitatively calculate the error between the reprojected corner points and the detected corner points. Through the experimental verification of the present invention, the reprojected points are basically consistent with the detected points, and the average error is within 10 pixels. The verification results are shown in the attached Figure 4 The second method is to develop a virtual camera simulation system in the form of software code, manually set the camera internal and external parameters in this simulation system and shoot "virtual" calibration data, put them into the calibration algorithm to calibrate the external parameters, and quantitatively measure the accuracy of the calibration results by comparing the errors between the calibrated external parameters and the set external parameters. The verification results of the virtual data set reprojection are shown in the attached Figure 5 .

[0109] The present invention has been verified by experiments to perform extrinsic calibration on multiple cameras of different viewing angles with only a two-dimensional chessboard as a calibration object, and the calibration results are sufficiently accurate. The method of the present invention focuses on the extrinsic calibration problem for camera systems arranged at multiple viewing angles, but the method of the present invention is still effective for multi-camera systems under large baseline conditions.

[0110] Based on the method provided by the present invention, the present invention also provides a multi-view camera calibration system based on nonlinear optimization, the system comprising:

[0111] An intrinsic parameter pre-calibration module is used to pre-calibrate the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrices of all cameras to be calibrated;

[0112] The external parameter rough calibration module is used to roughly calibrate the external parameters of all cameras to be calibrated in the selected world coordinate system through the neighboring relationship between multiple cameras, and obtain the initial external parameters to be optimized of all cameras to be calibrated;

[0113] The extrinsic parameter optimization module is used to synchronously optimize the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix through a nonlinear optimization method to obtain refined camera extrinsic parameters.

[0114] Wherein, the internal reference pre-calibration module specifically includes:

[0115] The intrinsic parameter pre-calibration unit is used to pre-calibrate the internal parameters of each camera to be calibrated separately using the CameraCalibration calibration toolbox built into the Matlab software to obtain the intrinsic parameter matrix of all cameras to be calibrated.

[0116] The external parameter rough calibration module specifically includes:

[0117] A camera coordinate calculation unit, used for selecting a camera coordinate system of any camera among all the cameras to be calibrated as a world coordinate system of the entire camera system, and calculating the coordinates of all the cameras to be calibrated in the world coordinate system;

[0118] An adjacent relationship determining unit, used to determine the adjacent relationship between all the cameras to be calibrated according to the coordinates of all the cameras to be calibrated in the world coordinate system;

[0119] An external parameter rough calibration unit, used for calibrating the external parameters of every two cameras among all the cameras to be calibrated according to the pairwise neighbor relationship between all the cameras to be calibrated;

[0120] The initial extrinsic parameter calculation unit is used to uniformly transfer the extrinsic parameters of all cameras to be calibrated to the world coordinate system by utilizing the pairwise adjacent relationship between all cameras to be calibrated, so as to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated in the world coordinate system.

[0121] The external parameter optimization module includes:

[0122] A calibration image acquisition unit, used to acquire chessboard images of different postures synchronously taken by all cameras to be calibrated;

[0123] A corner point detection unit, used to detect corner points on the chessboard images with different postures, and obtain corner points in all postures of the chessboard;

[0124] A reprojection unit, for reprojecting, for each chessboard posture, the corner points under the chessboard posture back to the corresponding chessboard image taken by each camera to be calibrated according to the intrinsic parameter matrix and the initial extrinsic parameter, to obtain a reprojection point;

[0125] A Euclidean distance calculation unit, used to calculate the Euclidean distance between the reprojection point and the corresponding corner point;

[0126] The extrinsic parameter optimization unit is used to minimize the Euclidean distance through the LM optimization algorithm to obtain refined camera extrinsic parameters.

[0127] The system further comprises:

[0128] The calibration result verification module is used to quantitatively measure the accuracy of the refined camera extrinsic calibration result by using a reprojection verification method.

[0129] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0130] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-view camera calibration method based on nonlinear optimization, characterized in that: include: Pre-calibrate the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated; The pre-calibration of the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrix of all cameras to be calibrated specifically includes: Use the CameraCalibration calibration toolbox built into the Matlab software to pre-calibrate the internal parameters of each camera to be calibrated, and obtain the internal parameter matrix of all cameras to be calibrated; The pre-calibration results are the radial distortion parameters (k1, k2) and the intrinsic parameter matrix of the camera. Where (f x ,f y ) represents the normalized focal length of the camera in the direction of the two image axes, (c x ,c y ) represents the coordinates of the camera principal point; K i is the set of intrinsic parameter matrices of all cameras to be calibrated; the parameters obtained in this stage will be used as constants in the projection calculation process of the subsequent extrinsic parameter calibration stage; The external parameters of all cameras to be calibrated in the selected world coordinate system are roughly calibrated through the neighboring relationship between multiple cameras, and the initial external parameters to be optimized of all cameras to be calibrated are obtained; The method of roughly calibrating the external parameters of all cameras to be calibrated in the selected world coordinate system through the neighboring relationship between multiple cameras to obtain the initial external parameters to be optimized of all cameras to be calibrated specifically includes: For a multi-camera system where all the cameras to be calibrated form a circle and focus on the same central area, select the camera coordinate system of any camera among all the cameras to be calibrated as the world coordinate system of the entire camera system, and calculate the coordinates of all the cameras to be calibrated in the world coordinate system; Determine the pairwise adjacent relationship between all the cameras to be calibrated according to the coordinates of all the cameras to be calibrated in the world coordinate system; Calibrate the external parameters of all the cameras to be calibrated according to the pairwise neighbor relationship between all the cameras to be calibrated; The extrinsic parameters of all cameras to be calibrated are uniformly transferred to the world coordinate system by utilizing the pairwise adjacent relationship between all cameras to be calibrated, so as to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated in the world coordinate system; By using a nonlinear optimization method, based on the intrinsic parameter matrix, the initial extrinsic parameters to be optimized of all cameras to be calibrated are simultaneously optimized to obtain refined camera extrinsic parameters; The nonlinear optimization method is used to synchronously optimize the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix to obtain refined camera extrinsic parameters, specifically including: Obtain chessboard images of different postures taken synchronously by all cameras to be calibrated; Detecting corner points on the chessboard images in different postures to obtain corner points in all postures of the chessboard; For each chessboard posture, reproject the corner points under the chessboard posture back to the corresponding chessboard image taken by each camera to be calibrated according to the intrinsic parameter matrix and the initial extrinsic parameter to obtain the reprojection points; Calculating the Euclidean distance between the reprojection point and the corresponding corner point; The Euclidean distance is minimized by the LM optimization algorithm to obtain refined camera extrinsics.

2. The method according to claim 1, characterized in that The method further comprises: The reprojection verification method is used to quantitatively measure the accuracy of the refined camera extrinsic calibration results.

3. A multi-view camera calibration system based on nonlinear optimization, characterized in that: include: An intrinsic parameter pre-calibration module is used to pre-calibrate the intrinsic parameters of all cameras to be calibrated to obtain the intrinsic parameter matrices of all cameras to be calibrated; The internal reference pre-calibration module specifically includes: The internal parameter pre-calibration unit is used to pre-calibrate the internal parameters of each camera to be calibrated separately using the CameraCalibration calibration toolbox built into the Matlab software to obtain the internal parameter matrix of all cameras to be calibrated; The pre-calibration results are the radial distortion parameters (k1, k2) and the intrinsic parameter matrix of the camera. Where (f x ,f y ) represents the normalized focal length of the camera in the direction of the two image axes, (c x ,c y ) represents the coordinates of the camera principal point; K i is the set of intrinsic parameter matrices of all cameras to be calibrated; the parameters obtained in this stage will be used as constants in the projection calculation process of the subsequent extrinsic parameter calibration stage; The external parameter rough calibration module is used to roughly calibrate the external parameters of all cameras to be calibrated in the selected world coordinate system through the neighboring relationship between multiple cameras, and obtain the initial external parameters to be optimized of all cameras to be calibrated; The external parameter rough calibration module specifically includes: A camera coordinate calculation unit is used to select the camera coordinate system of any camera among all the cameras to be calibrated as the world coordinate system of the entire camera system for a multi-camera system in which all the cameras to be calibrated form a circle and focus on the same central area, and calculate the coordinates of all the cameras to be calibrated in the world coordinate system; An adjacent relationship determining unit, used to determine the adjacent relationship between all the cameras to be calibrated according to the coordinates of all the cameras to be calibrated in the world coordinate system; An external parameter rough calibration unit, used for calibrating the external parameters of every two cameras among all the cameras to be calibrated according to the pairwise neighbor relationship between all the cameras to be calibrated; An initial extrinsic parameter calculation unit is used to uniformly transfer the extrinsic parameters of all cameras to be calibrated to the world coordinate system by utilizing the pairwise adjacent relationship between all cameras to be calibrated, so as to obtain the initial extrinsic parameters to be optimized of all cameras to be calibrated in the world coordinate system; An extrinsic parameter optimization module, used to synchronously optimize the initial extrinsic parameters to be optimized of all cameras to be calibrated based on the intrinsic parameter matrix by a nonlinear optimization method, so as to obtain refined camera extrinsic parameters; The external parameter optimization module includes: A calibration image acquisition unit, used to acquire chessboard images of different postures synchronously taken by all cameras to be calibrated; A corner point detection unit, used to detect corner points on the chessboard images with different postures, and obtain corner points in all postures of the chessboard; A reprojection unit, for reprojecting, for each chessboard posture, the corner points under the chessboard posture back to the corresponding chessboard image taken by each camera to be calibrated according to the intrinsic parameter matrix and the initial extrinsic parameter, to obtain a reprojection point; A Euclidean distance calculation unit, used to calculate the Euclidean distance between the reprojection point and the corresponding corner point; The extrinsic parameter optimization unit is used to minimize the Euclidean distance through the LM optimization algorithm to obtain refined camera extrinsic parameters.

4. The system according to claim 3, characterized in that The system further comprises: The calibration result verification module is used to quantitatively measure the accuracy of the refined camera extrinsic calibration result by using a reprojection verification method.

Images