Multi-camera extrinsic parameter calibration method, system, medium and terminal based on mobile calibration plate

Abstract

The present invention provides a multi-camera extrinsic calibration method based on a mobile calibration plate, comprising: a calibration vehicle equipped with a calibration plate moves within the field of view of multiple RGB-D cameras; obtaining the checkerboard corner points in each camera image, solving the pose of the checkerboard and each camera; based on the pose, establishing a pose graph of the checkerboard and all cameras, and optimizing the pose graph; based on the optimized pose graph, obtaining the global extrinsic calibration of all cameras. The present invention establishes a pose graph of multiple cameras and the checkerboard, solves and optimizes, obtains the optimal camera pose, and improves the calibration accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of camera extrinsic calibration in the field of image processing, and in particular to a multi-camera extrinsic calibration method, system and terminal based on a mobile calibration plate. Background Art

[0002] Automated Valet Parking (AVP) is a hot topic in current research. Compared with the single-vehicle intelligent parking solution, the intelligent parking solution for the parking lot has the advantages of rich global information and interactive communication between parking lots. Low-cost RGB-D cameras have promoted the implementation of the intelligent AVP solution for the parking lot.

[0003] Distributed RGB-D cameras capture color and depth images of indoor parking lots from different positions and angles, reconstructing the color and geometry of the scene. Accurate and fast calibration of camera extrinsics (translation and rotation) is a fundamental step in building a camera network that can capture large scenes and merge multiple camera views into the same coordinate system.

[0004] Generally speaking, the external parameter calibration methods for multiple cameras can be divided into methods that use calibration objects and methods that do not use calibration objects. The methods that do not use calibration objects are highly dependent on the scene and have an inaccurate understanding of the unknown scene features, resulting in low calibration accuracy and many incorrect correspondences; the use of calibration objects such as calibration plates and calibration balls requires the design of a stable and accurate detection algorithm, as well as a better error function.

[0005] Although many calibration objects have been used to calibrate multiple cameras, most of them are not suitable for indoor parking environments where cameras are sparsely placed. In addition, when the number of cameras is large, the calibration error and calibration complexity will increase significantly. Summary of the invention

[0006] In view of the defects in the prior art, an object of the present invention is to provide a multi-camera extrinsic parameter calibration method and system based on a mobile calibration plate.

[0007] According to one aspect of the present invention, a multi-camera extrinsic calibration method based on a mobile calibration plate is provided, wherein a calibration vehicle equipped with a calibration plate moves within the field of view of multiple RGB-D cameras;

[0008] Get the checkerboard corner points in each camera image and solve the pose of the checkerboard and a single camera;

[0009] Based on the poses of the chessboard and a single camera, a pose graph of the chessboard and all cameras is established, and the pose graph is optimized;

[0010] Based on the optimized pose graph, the global extrinsic calibration of all cameras is obtained.

[0011] Preferably, when moving into a common field of view of multiple cameras, the stay time of the calibration vehicle is not less than 10 seconds.

[0012] Preferably, the obtaining of checkerboard corner points in each camera image and solving the pose of the checkerboard and a single camera comprises:

[0013] Use the findChessboardCornerSB() method in the OpenCV4 image processing library to detect all corner points of the chessboard in the image plane;

[0014] Based on all corner points, solve the PnP (Perspective-n-Point) problem and obtain the relative position of the chessboard and the camera.

[0015]

[0016] Where: K C is the camera internal parameter, d C is the camera distortion parameter, B′ is the position of all corner points of the chessboard detected in the image, B B is the position of all corner points of the chessboard in space;

[0017] Define the reprojection error function:

[0018]

[0019] Where: (r,c)∈I is the index of the corner point of the chessboard with length and width r and c, b′ r,c is the position of the corner point detected in the image plane, B b r,c is the position of each point on the chessboard in the chessboard coordinate system, rep C (·) is the reprojection function of a 3D point in space onto the image plane calculated by the intrinsic parameters and distortion parameters;

[0020] The reprojection error function is minimized to obtain the internal parameters of a single camera, that is, the position and posture of the chessboard and the single camera are obtained.

[0021] Preferably, the step of establishing a pose graph of the chessboard and all cameras based on the poses of the chessboard and a single camera, and optimizing the pose graph comprises:

[0022] Construct a pose graph, the vertices of which store the poses of the camera and the calibration plate in the global coordinate system, and each edge is used to store the relative pose transformation between the camera vertex and the calibration plate vertex

[0023] The pose constraints in the position graph are converted into error functions:

[0024]

[0025] In the formula, is the observation coefficient. In the calibration stage k, if the camera C i If a checkerboard is detected, then otherwise is the regularization factor, indicating that the camera C i The error of the estimated position of the corner point in the provided image is usually 0.5 or 1. The reprojection error function It can be expanded into a chessboard corner point representation form, and the chessboard is reprojected to the pixel plane, and the difference with the detected corner point is obtained to obtain the projection error; The camera pose is and is the chessboard posture.

[0026] The error function is minimized and solved using the Levenberg-Marquardt method, and the Ceres Solver nonlinear optimization library is used for calculation to obtain an optimized pose graph.

[0027] Preferably, obtaining a global pose estimation of the camera based on the optimized pose graph includes:

[0028] Based on the global coordinate system assumed by the ground plane, the calibration plate is placed at different positions on the ground to obtain the posture of the chessboard relative to the ground, which is a two-dimensional posture constraint of the plane, and is determined by the translation amount (t x ,t y ) and the rotation amount θ:

[0029]

[0030] The two-dimensional constraint of the chessboard's posture is added to the error function, and the posture relationship of the calibration plate is transformed to:

[0031]

[0032] The above expression converts the pose of the calibration plate relative to the world coordinate system into a two-dimensional plane pose and the pose of the calibration plate relative to the two-dimensional plane.

[0033] Calculate and obtain the external parameters of multiple RGB-D cameras, that is, obtain the global external parameter calibration of all cameras.

[0034] According to a second aspect of the present invention, a multi-camera extrinsic parameter calibration method based on a mobile calibration plate is provided, comprising:

[0035] A calibration vehicle, wherein the calibration vehicle is equipped with a wifi module, a motor drive and a controller,

[0036] A calibration plate, which is mounted on the top of the calibration vehicle and moves with it;

[0037] RGB-D cameras, wherein the plurality of RGB-D cameras are distributed in a plurality of positions and are used to photograph the calibration plate;

[0038] A switch, wherein the switch receives data information of the RGB-D camera;

[0039] A server receives data information processed by the switch.

[0040] Preferably, the RGB-D camera can obtain color images and depth images, wherein the depth image can be calculated and converted into point cloud information.

[0041] Preferably, the edge computing unit transmits the data of multiple RGB-D cameras to the server for calculation and analysis.

[0042] The Chrony time synchronization method is used to synchronize the time of multiple edge computing units, and the server time is used as the benchmark.

[0043] According to a third aspect of the present invention, there is provided a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods described, or to run any one of the systems described when executing the program.

[0044] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described, or to run any of the systems described.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The distributed multi-RGB-D camera extrinsic calibration method based on pose graph optimization of the present invention establishes pose graphs of multiple cameras and a chessboard, solves and optimizes them, obtains the optimal camera pose, and improves the calibration accuracy;

[0047] (2) The present invention can realize automatic calibration of external parameters of multiple RGB-D cameras on the field side, with low cost, simple operation, reduced manual participation, and low computational complexity;

[0048] (3) The distributed multi-RGB-D camera extrinsic calibration system based on pose graph optimization of the present invention broadens the solution ideas for problems in similar fields. This system is the first to be implemented in this field and has high research value.

[0049] (4) The distributed multi-RGB-D camera extrinsic calibration system based on pose graph optimization invented in this paper has the function of remote image transmission, which helps users intuitively perceive the vehicle position, remotely control the vehicle movement function, and move the calibration plate to different positions within the camera field of view, reducing the time and physical effort consumed by manually moving the calibration plate, thereby improving the calibration efficiency; BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0051] Figure 1 A schematic diagram of a multi-camera extrinsic calibration method based on a mobile calibration plate according to an embodiment of the present invention;

[0052] Figure 2 A system design diagram of a mobile calibration board according to an embodiment of the present invention, which is composed of a user terminal and a calibration vehicle;

[0053] Figure 3 A flowchart of distributed RGB-D camera extrinsic calibration according to an embodiment of the present invention;

[0054] Figure 4 An abstract diagram of a camera and a calibration plate pose diagram according to an embodiment of the present invention;

[0055] Figure 5 This is a global extrinsic calibration of all cameras according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0057] like Figure 1 FIG. 1 is a flowchart of a multi-camera extrinsic calibration method based on a mobile calibration plate according to an embodiment of the present invention, comprising:

[0058] S100, the calibration car equipped with the calibration plate moves within the field of view of multiple RGB-D cameras;

[0059] S200, obtaining the corner points of the chessboard in each camera image, and solving the pose of the chessboard and each camera;

[0060] S300, based on the posture, establishing a posture graph of the chessboard and all cameras, and optimizing the posture graph;

[0061] S400, based on the optimized pose graph, obtains the global extrinsic calibration of all cameras.

[0062] Based on the above embodiments, the present invention provides a preferred embodiment, such as Figure 2 As shown, it is a flow chart of the multi-camera extrinsic calibration method based on a mobile calibration plate in this embodiment. In this embodiment, the field-side distributed multi-RGB-D camera system is used as a platform to analyze and calculate the acquired images, and explore how to efficiently and simply calibrate the extrinsic parameters between cameras. The calibration object used is the calibration plate, and the calibration algorithm is mainly a multi-RGB-D camera extrinsic calibration method based on pose graph optimization.

[0063] In view of the complexity of external parameter calibration of the distributed RGB-D camera at the field end, this embodiment designs a mobile calibration plate as a calibration reference object, such as Figure 3 As shown in the figure, it includes a calibration vehicle and a user operation interface. The user's mobile phone is connected to the calibration vehicle through local WIFI technology. The camera on the calibration vehicle obtains images and transmits them back to the user. The calibration vehicle is equipped with a calibration plate on the top of the vehicle, which moves within the camera's field of view and associates the poses between cameras. A calibration algorithm is designed to process the images obtained by the camera and solve the external parameters between cameras, in preparation for the subsequent use of multiple RGB-D cameras on the field.

[0064] The multi-camera extrinsic calibration method based on the mobile calibration plate in this embodiment includes:

[0065] S11, start multiple cameras on the central server through remote login;

[0066] S12, obtains real-time image data from multiple RGB-D cameras, collects image data through a low-cost edge computing unit, such as Raspberry Pi, compresses the image data, and transmits it to the central server through the network to reduce network transmission bandwidth;

[0067] S13, operates the calibration vehicle to move within the camera field of view and stays longer at key positions such as the common field of view of the cameras to obtain more effective constraints between the cameras, and then detects the checkerboard corner points in each camera image.

[0068] In order to obtain the chessboard corner points more completely, a preferred embodiment is provided, using the findChessboardCornerSB() method in the OpenCV4 image processing library to detect all the corner points of the chessboard in the image plane;

[0069] S14, camera chessboard pose estimation, when the complete chessboard corner points are detected, it is necessary to estimate the relative pose of the chessboard and the camera.

[0070] In order to more accurately obtain the relative position of the chessboard and the camera, a preferred embodiment is provided, and the corresponding solution is obtained by solving the PnP (Perspective-n-Point) problem:

[0071]

[0072] Where: K C is the camera internal parameter, d C is the camera distortion parameter, B′ is the position of all corner points of the chessboard detected in the image, B B is the position of all corner points of the chessboard in space.

[0073] In order to solve the above PnP problem, it is necessary to define the reprojection error function:

[0074]

[0075] Where: (r,c)∈I is the index of the corner point of the chessboard with length and width r and c, b′ r,c is the position of the corner point detected in the image plane, B b r,c is the position of each point on the chessboard in the chessboard coordinate system, rep C (·) is the reprojection function of a 3D point in space onto the image plane, which is calculated using the intrinsic parameters and distortion parameters.

[0076] S15, establishment of pose graph, the mobile calibration vehicle is in the public field of view. After obtaining the relative pose of the camera and the chessboard, more pose data are added during the calibration process. It is necessary to process the relationship between these data and provide constraints for the accuracy of the external parameter calibration.

[0077] In order to make the establishment of the pose graph more accurate, the present invention provides a preferred embodiment, such as Figure 3 As shown in the figure, it is an abstract graph of the pose graph of the camera and the calibration plate. The vertices in the pose graph are used to store the poses of the camera and the calibration plate in the global coordinate system, and each edge is used to store the relative pose transformation between the camera vertex and the calibration plate vertex.

[0078] S16, pose graph solution, after the pose graph is established, the pose constraints need to be converted into an error function for numerical calculation. Further, the error function is as follows:

[0079]

[0080] In the formula, is the observation coefficient. At calibration stage k (calibration stage k represents time, kth frame image), if camera C i If a checkerboard is detected, then otherwise is the regularization factor, indicating that the camera C i The error of the estimated position of the corner point in the provided image is usually 0.5 or 1. The reprojection error function It can be expanded into a chessboard corner point representation form, and the chessboard is reprojected to the pixel plane, and the difference with the detected corner point is obtained to obtain the projection error; The camera pose is and is the chessboard posture.

[0081] In order to obtain the optimal multi-camera pose, the error function needs to be minimized.

[0082] Furthermore, the Levenberg-Marquardt (LM) method is used to solve the problem, and the Ceres Solver nonlinear optimization library is used for computational analysis. By establishing a pose graph of multiple cameras and a chessboard, the relative poses between multiple cameras are associated, and by collecting multiple frames of data, the errors in the calibration phase are accumulated, and the external parameters between multiple cameras are obtained by finding the global optimal solution.

[0083] S17, based on the global coordinate system extraction of the ground plane assumption, the calibration plate is placed at different positions on the ground. Then, relative to the ground, the relative posture of the chessboard is expressed as:

[0084]

[0085] The chessboard becomes a two-dimensional motion, and the translation (t x ,t y ) and the rotation θ, and then the two-dimensional constraint of the chessboard posture is added to the error function, and the posture relationship of the calibration plate is transformed into:

[0086]

[0087] The above expression converts the position and posture of the calibration plate relative to the world coordinate system into a two-dimensional plane posture and the position and posture of the calibration plate relative to the two-dimensional plane, which reduces the number of parameters and improves the calculation efficiency.

[0088] In the actual calibration process, in order to find the ground as the world coordinate system, it is often inappropriate to use a large area of ​​calibration plate as a calibration reference. Through the global coordinate extraction method based on the ground plane assumption in this article, it is only necessary to place calibration plates at multiple locations on the ground and constrain multiple checkerboards to be on the same plane in the algorithm, so that the global ground extraction will have a higher consistency.

[0089] S18, calculate and obtain the external parameters of multiple RGB-D cameras and save them, such as Figure 5 As shown, the global extrinsic calibration results of all cameras in this embodiment are displayed, which are composed of camera pose, chessboard pose and ground.

[0090] In other embodiments of the present invention, Figure 1 The calibration process and Figure 3The multi-camera extrinsic parameter calibration based on the pose graph optimization method is performed on ten RGB-D cameras installed at the indoor parking lot. The data is transmitted to the central server through the Raspberry Pi. The mobile calibration car is placed in the camera field of view, the camera poses are associated, the camera and chessboard pose graphs are established, and the error function is optimized regularly to solve the multi-RGB-D camera extrinsic parameters. The actual calibration results are shown in the figure. Figure 4 shown.

[0091] Based on the same concept of the above embodiment, the present invention provides another embodiment. A multi-camera extrinsic parameter calibration method based on a mobile calibration plate comprises: a calibration vehicle, a calibration plate, an RGB-D camera, a switch and a server, wherein the calibration vehicle is equipped with a wifi module, a motor drive and a controller, and the calibration plate is installed on the top of the calibration vehicle and moves with it; the multiple RGB-D cameras are distributed in multiple directions to shoot the calibration plate; the switch receives data information of the RGB-D camera; and the server receives data information processed by the switch.

[0092] Furthermore, the RGB-D camera can obtain color images and depth images, where the depth map can be calculated and converted into point cloud information.

[0093] Based on the same concept of the above embodiments, in other embodiments of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above methods or to run any of the systems when executing the program.

[0094] Based on the same concept of the above embodiments, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute any of the above methods, or run any of the above systems.

[0095] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A multi-camera extrinsic calibration method based on a mobile calibration plate, characterized in that: A calibration car equipped with a checkerboard moves within the field of view of multiple RGB-D cameras; Get the checkerboard corner points in each camera image and solve the pose of the checkerboard and a single camera; Based on the poses of the chessboard and a single camera, a pose graph of the chessboard and all cameras is established, and the pose graph is optimized; Based on the optimized pose graph, obtain the global extrinsic calibration of all cameras; The step of obtaining the checkerboard corner points in each camera image and solving the pose of the checkerboard and a single camera includes: Use the findChessboardCornerSB() method in the OpenCV4 image processing library to detect all corner points of the chessboard in the image plane; Based on all the corner points, solve the PnP (Perspective-n-Point) problem and obtain the relative position of the chessboard and the camera. Where: K C is the camera internal parameter, d C is the camera distortion parameter, B′ is the position of all corner points of the chessboard detected in the image, B B is the position of all corner points of the chessboard in space; Define the reprojection error function, Where: (r,c)∈I is the index of the corner point of the chessboard with length and width r and c, b r ' ,c is the position of the corner point detected in the image plane, B b r,c is the position of each point on the chessboard in the chessboard coordinate system, rep C (·) is the reprojection function of a 3D point in space onto the image plane calculated by the intrinsic parameters and distortion parameters; Minimize the reprojection error function to obtain the internal parameters of a single camera, that is, obtain the position and posture of the chessboard and the single camera; The step of establishing a pose graph of the chessboard and all cameras based on the pose of the chessboard and a single camera, and optimizing the pose graph includes: Construct a pose graph, the vertices of which store the poses of the camera and the calibration plate in the global coordinate system, and each edge is used to store the relative pose transformation between the camera vertex and the chessboard vertex The pose constraints in the pose graph are converted into an error function: In the formula, is the observation coefficient. In the calibration stage k, if the camera C i If a checkerboard is detected, then otherwise is the regularization factor, indicating that camera C i The error of the estimated position of the corner point in the provided image is 0.5 or 1, and the reprojection error function Expand it into a chessboard corner point representation, reproject the chessboard into the pixel plane, and obtain the projection error by comparing it with the detected corner point. The camera pose is and is the chessboard position; Minimize the error function, solve it using the Levenberg-Marquardt method, and calculate it using the Ceres Solver nonlinear optimization library to obtain an optimized pose graph; The method of obtaining the global pose estimation of the camera based on the optimized pose graph includes: Based on the global coordinate system assumed by the ground plane, the calibration plate is placed at different positions on the ground to obtain the posture of the chessboard relative to the ground, which is a two-dimensional posture constraint of the plane, and is determined by the translation amount (t x ,t y ) and the rotation amount θ: The two-dimensional constraint of the chessboard's posture is added to the error function, and the posture relationship of the calibration plate is transformed to: The position of the calibration plate relative to the world coordinate system Converted into the calibration plate relative to the two-dimensional plane pose The error function is optimized and multiple RGB-D camera extrinsic parameters are calculated, that is, the global extrinsic parameter calibration of all cameras is obtained.

2. The multi-camera extrinsic calibration method based on a mobile calibration plate according to claim 1, characterized in that: When moving into the common field of view of multiple cameras, the stay time of the calibration vehicle is not less than 10 seconds.

3. A system for implementing the field-side multi-RGB-D camera extrinsic calibration method of any one of claims 1-2 using a mobile calibration plate, characterized in that: include: A calibration vehicle, wherein the calibration vehicle is equipped with a wifi module, a motor drive and a controller; A calibration plate, which is mounted on the top of the calibration vehicle and moves with it; RGB-D cameras, wherein the plurality of RGB-D cameras are distributed in a plurality of positions and are used to photograph the calibration plate; A switch, wherein the switch receives data information of the RGB-D camera; A server receives data information processed by the switch.

4. The multi-camera extrinsic calibration system based on a mobile calibration plate according to claim 3, characterized in that: The RGB-D camera can obtain color images and depth images, wherein the depth image is converted into point cloud information through calculation.

5. The multi-camera extrinsic calibration system based on a mobile calibration plate according to claim 3, characterized in that: The edge computing unit transmits data of multiple RGB-D cameras to the server for calculation and analysis, and uses the Chrony time synchronization method to synchronize the time of multiple edge computing units, with the server time as the benchmark.

6. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1-2, or to run the system described in any one of claims 3-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 2, or to run the system described in any one of claims 3 to 5.

Images