Method, device, computer equipment and medium for calibrating external parameters of a multi-camera system

Abstract

The present application provides a method, apparatus, computer device, and medium for calibrating the external parameters of a multi-camera system. The method includes: obtaining multiple frames of images collected synchronously by multiple cameras for a cross-marker linkage; a plurality of reflective markers are arranged on the cross-marker linkage; based on the multiple frames of images of the cross-marker linkage, calculating the three-dimensional coordinates of the plurality of reflective markers on the cross-marker linkage in the camera coordinate system; according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system, calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system; using an optimization algorithm to process the rotation matrix and translation vector to obtain the rotation matrix and translation vector of each camera to the reference coordinate system after processing. The external parameter calibration method described in the present application can improve the accuracy and robustness of the calibration results of multiple cameras to the reference coordinate system.

Description

Technical Field

[0001] This application relates to the technical field of camera calibration, and more specifically, to an external parameter calibration method, device, computer device, and medium for a multi-camera system. Background Art

[0002] Optical motion capture technology is widely used in fields such as film and television production and entertainment interaction. With its advantages of high precision and convenient operation and control, it has been widely applied in recent years. Among them, the multi-camera calibration technology in the optical motion capture system is the cornerstone for the entire system to operate with high precision and high stability. The quality of the camera calibration results determines whether the target space coordinate position can be accurately identified, which in turn affects subsequent interactive extension operations of the system.

[0003] Multi-camera calibration is divided into internal parameter calibration and external parameter calibration. Internal parameter calibration is to calibrate the intrinsic parameters of each camera through a certain calibration object, such as the internal parameter matrix and distortion coefficient, etc.; external parameter calibration can be divided into camera-to-camera attitude calibration and multi-camera-to-reference coordinate system attitude calibration according to the process. When an optical motion capture system completes the internal and external parameter calibration of multiple cameras, the spatial coordinates of the target points in the overlapping field of view of multiple cameras can be calculated through technologies such as triangulation, and then extended to other operations, such as attitude recognition and motion tracking.

[0004] Camera calibration can be divided into one-dimensional calibration, two-dimensional calibration, and three-dimensional calibration according to the form of the calibration object. In the prior art, using a one-dimensional calibration object and a calibration algorithm to calibrate the camera internal parameters with high precision and the external parameters between cameras have been applied relatively maturely. However, for the external parameter calibration process of multi-camera-to-reference coordinate system attitude calibration, it is difficult to obtain calibration results with both high precision and robustness. Summary of the Invention

[0005] In view of this, the embodiments of this application provide an external parameter calibration method, device, computer device, and medium for a multi-camera system, which can improve the accuracy and robustness of the calibration results of multi-camera-to-reference coordinate system.

[0006] In a first aspect, the embodiments of this application provide an external parameter calibration method for a multi-camera system, including the following steps:

[0007] Obtain multiple frames of images collected synchronously by multiple cameras of a cross-marker linkage; a plurality of reflective markers are arranged on the cross-marker linkage;

[0008] Based on the multiple frames of images of the cross-marker linkage, calculate the three-dimensional coordinates of the plurality of reflective markers on the cross-marker linkage in the camera coordinate system;

[0009] Calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system;

[0010] Process the rotation matrix and translation vector using an optimization algorithm to obtain the processed rotation matrix and translation vector from each camera to the reference coordinate system.

[0011] In a possible implementation, calculating the three-dimensional coordinates of the multiple reflective markers on the cross-marker link in the camera coordinate system based on the multiple frames of images of the cross-marker link includes:

[0012] Parse the multiple frames of images of the cross-marker link to determine the three-dimensional coordinates in the camera coordinate system of the first number of reflective markers that are collinear the most among the line segments formed by the multiple reflective markers;

[0013] Based on the three-dimensional coordinates of the first number of reflective markers in the camera coordinate system, obtain the three-dimensional coordinates of the remaining reflective markers in the camera coordinate system.

[0014] In a possible implementation, after the step of calculating the three-dimensional coordinates of the multiple reflective markers on the cross-marker link in the camera coordinate system based on the multiple frames of images of the cross-marker link, the method further includes:

[0015] Process the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system using an optimization algorithm to obtain the processed three-dimensional coordinates;

[0016] The calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system includes:

[0017] Calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the processed three-dimensional coordinates of the multiple reflective markers in the camera coordinate system.

[0018] In a possible implementation, the calculating the rotation matrix from the reference coordinate system to the camera coordinate system based on the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system includes:

[0019] Based on the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system, translate the origin of the reference coordinate system to the origin of the camera coordinate system to obtain a new coordinate system;

[0020] Based on the new coordinate system, calculate the unit direction vectors of the three coordinate axes of the new coordinate system;

[0021] Calculate the direction cosine vectors of the three coordinate axes of the newly established coordinate system based on the unit direction vectors of the three coordinate axes of the newly established coordinate system and the unit direction vectors of the respective reference coordinate axes of the camera coordinate system;

[0022] Obtain the rotation matrix from the reference coordinate system to the camera coordinate system according to the direction cosine vectors of the three coordinate axes of the newly established coordinate system.

[0023] In a possible implementation manner, before the step of processing the rotation matrix and the translation vector by using the optimization algorithm, the method further includes:

[0024] Orthogonalize the rotation matrix to obtain the initial value of the rotation matrix from the reference coordinate system to the camera coordinate system.

[0025] In a possible implementation manner, the step of processing the rotation matrix and the translation vector by using the optimization algorithm to obtain the processed rotation matrix and translation vector from each camera to the reference coordinate system includes:

[0026] Obtain the initial internal parameters of each camera and the pixel coordinates of the reflective markers;

[0027] Based on the initial internal parameters of each camera, the rotation matrix and translation vector from each camera to the reference coordinate system, and the pixel coordinates and three-dimensional coordinates of the reflective markers in the camera coordinate system, obtain the processed initial internal parameters of each camera and the rotation matrix and translation vector from each camera to the reference coordinate system through the optimization algorithm.

[0028] In a possible implementation manner, there are at least four reflective markers provided on the cross marker link, and the distances between adjacent reflective markers are all different;

[0029] The line segments formed by sequentially connecting all the reflective markers are two mutually perpendicular line segments.

[0030] In a second aspect, an external parameter calibration device for a multi-camera system provided by an embodiment of the present application is applied to the external parameter calibration of at least two cameras, and includes:

[0031] An acquisition module, configured to acquire multiple frames of images obtained by multiple cameras synchronously collecting a cross marker link; a plurality of reflective markers are provided on the cross marker link;

[0032] A first calculation module, configured to calculate the three-dimensional coordinates of the plurality of reflective markers on the cross marker link in the camera coordinate system based on the multiple frames of images of the cross marker link;

[0033] A second calculation module, configured to calculate a rotation matrix and a translation vector from a reference coordinate system to the camera coordinate system according to three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system;

[0034] A processing module, configured to process the rotation matrix and the translation vector by using an optimization algorithm to obtain processed rotation matrices and translation vectors of each camera to the reference coordinate system.

[0035] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-camera system external parameter calibration method according to any one of the first aspects are implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the multi-camera system external parameter calibration method according to any one of the first aspects are executed.

[0037] The technical solution provided by the embodiment of the present application has the following beneficial effects:

[0038] For the external parameter calibration method of the present application, in order to obtain the three-dimensional coordinates of the reflective markers on the cross marker link, it is necessary to first obtain multiple frames of images collected by multiple cameras synchronously for the cross marker link. A plurality of reflective markers are arranged on the cross marker link. Then, based on the multiple frames of images of the cross marker link, the three-dimensional coordinates of the plurality of reflective markers on the cross marker link in the camera coordinate system are calculated, and according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system, a rotation matrix and a translation vector from the reference coordinate system to the camera coordinate system are calculated. Finally, in order to obtain a more accurate and robust attitude relationship of each camera to the reference coordinate system, an optimization algorithm is used to process the rotation matrix and the translation vector to obtain processed rotation matrices and translation vectors of each camera to the reference coordinate system. For the external parameter calibration method of the present application, by calculating the three-dimensional coordinates of the reflective markers on the cross marker link, calculating a rotation matrix according to the three-dimensional coordinates, and finally optimizing and processing the rotation matrix and the translation vector by an optimization algorithm, the obtained attitude relationship of each camera to the reference coordinate system is made more accurate, that is, the accuracy and robustness of the external parameter calibration of each camera are improved.

[0039] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flowchart of a method for calibrating the external parameters of a multi-camera system provided by an embodiment of the present application;

[0042] Figure 2 It is a schematic flowchart of a method for determining the three-dimensional coordinates of a reflective marker provided by an embodiment of the present application;

[0043] Figure 3 It is a schematic structural diagram of the cross-marker connecting rod provided by an embodiment of the present application;

[0044] Figure 4 It is a schematic flowchart of a method for calculating a rotation matrix provided by an embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of the relationship between the newly established coordinate system, the reference coordinate system, and the camera coordinate system provided by an embodiment of the present application;

[0046] Figure 6 It is a schematic structural diagram of a device for calibrating the external parameters of a multi-camera system provided by an embodiment of the present application;

[0047] Figure 7 It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of them. Usually, the components of the embodiments of the present application described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0049] In the following description, "some embodiments" are involved, which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0050] An embodiment of the present application provides an external parameter calibration method for a multi-camera system, as Figure 1 shown, including the following steps:

[0051] S101, obtaining multiple frames of images obtained by synchronously collecting a cross-marker linkage by multiple cameras; a plurality of reflective markers are arranged on the cross-marker linkage;

[0052] S102, based on the multiple frames of images of the cross-marker linkage, calculating three-dimensional coordinates of the plurality of reflective markers on the cross-marker linkage in the camera coordinate system;

[0053] S103, according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system, calculating a rotation matrix and a translation vector from the reference coordinate system to the camera coordinate system;

[0054] S104, using an optimization algorithm to process the rotation matrix and the translation vector to obtain the rotation matrix and the translation vector from each camera to the reference coordinate system after processing.

[0055] The above exemplary steps of the embodiment of the present application will be described below.

[0056] In step S101, obtaining multiple frames of images obtained by synchronously collecting a cross-marker linkage by multiple cameras; a plurality of reflective markers are arranged on the cross-marker linkage;

[0057] In some embodiments, the cross-marker linkage may be an L-shaped calibration rod or a cross-shaped calibration rod; there are no less than four reflective markers arranged on the cross-marker linkage, and the distances between adjacent reflective markers are all different, and the line segments formed by sequentially connecting all the reflective markers are two mutually perpendicular line segments.

[0058] As an example, the cross-marker linkage described in the present application takes an L-shaped calibration rod as an example, and its structure is as Figure 3 shown, with four reflective markers arranged, and the distances between every two adjacent reflective markers are different, and the straight lines where the line segments formed by connecting the reflective markers are perpendicular to each other, as Figure 3 the straight line L in 012 、L 03Perpendicular to each other; when the cross - marker link is in a cross shape, it is also necessary to ensure that the reflective markers can be connected in sequence to form perpendicular line segments; in addition, it should be noted that the reflective markers are infrared - reflective markers. For the convenience of acquisition, the reflective markers can be spherical objects with infrared - reflective properties. When the camera emits infrared rays and projects them onto the surface of the reflective markers, due to the reflective properties of the reflective markers, most of the light can be reflected back to the camera, so that the coordinate information of the reflective markers can be easily extracted from the camera image.

[0059] In step S102, based on multiple frames of images of the cross - marker link, the three - dimensional coordinates of multiple reflective markers on the cross - marker link in the camera coordinate system are calculated.

[0060] Specifically, first, determine the reflective markers with the maximum collinear number, then calculate the three - dimensional coordinates of the reflective markers with the maximum collinear number in the camera coordinate system, and based on the calculated three - dimensional coordinates, obtain the three - dimensional coordinates of the remaining reflective markers in the camera coordinate system.

[0061] In step S103, according to the three - dimensional coordinates of the multiple reflective markers in the camera coordinate system, the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system are calculated.

[0062] Specifically, the translation vector from the reference coordinate system to the camera coordinate system is the coordinate of the intersection point of the L - shaped calibration rod. The acquisition of the translation vector is a prior art and will not be elaborated here; for the calculation of the rotation matrix, this application introduces a new coordinate system and obtains the unit direction vectors of the three coordinate axes of the new coordinate system. By jointly solving the unit direction vectors of the three coordinate axes of the new coordinate system and the unit direction vectors of the respective reference coordinate axes of the camera coordinate system, the direction cosine vectors of the three coordinate axes of the new coordinate system are obtained, and then a matrix is formed by the direction cosine vectors of the three coordinate axes of the new coordinate system, which is the rotation matrix from the reference coordinate system to the camera coordinate system.

[0063] In step S104, an optimization algorithm is used to process the rotation matrix and translation vector to obtain the processed rotation matrix and translation vector from each camera to the reference coordinate system.

[0064] In some embodiments, the optimization algorithm can be the Levenberg - Marquardt algorithm, which jointly performs non - linear optimization on the rotation matrix and translation vector. After optimization, the accurate internal parameters of each camera and the pose relationship from each camera to the reference coordinate system are obtained, thereby improving the accuracy and robustness of the external parameter calibration of multiple cameras. The pose relationship from each camera to the reference coordinate system is the rotation matrix and translation vector from each camera to the reference coordinate system.

[0065] For the above multi-camera system extrinsic parameter calibration method, in order to obtain the three-dimensional coordinates of the reflective markers on the cross-marker link, it is necessary to first obtain multiple frames of images of the cross-marker link collected synchronously by multiple cameras. There are multiple reflective markers on the cross-marker link. Then, based on the multiple frames of images of the cross-marker link, calculate the three-dimensional coordinates of the multiple reflective markers on the cross-marker link in the camera coordinate system, and calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system according to the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system. Finally, in order to obtain a more accurate and robust attitude relationship of each camera to the reference coordinate system, an optimization algorithm is used to process the rotation matrix and translation vector to obtain the processed rotation matrix and translation vector of each camera to the reference coordinate system. In the extrinsic parameter calibration method of the present application, by calculating the three-dimensional coordinates of the reflective markers on the cross-marker link, calculating the rotation matrix according to the three-dimensional coordinates, and finally optimizing the rotation matrix and translation vector through an optimization algorithm, the obtained attitude relationship of each camera to the reference coordinate system is more accurate, that is, the accuracy and robustness of the extrinsic parameter calibration of each camera are improved.

[0066] In some embodiments, in order to obtain the three-dimensional coordinates of multiple reflective markers in the camera coordinate system based on the multiple frames of images of the cross-marker link, the following operation process needs to be carried out. Specifically, as Figure 2 shown, step S102 includes the following steps:

[0067] S201, analyze the multiple frames of images of the cross-marker link to determine the three-dimensional coordinates of the first number of reflective markers with the most collinear in the line segment composed of multiple reflective markers in the camera coordinate system;

[0068] S202, based on the three-dimensional coordinates of the first number of reflective markers in the camera coordinate system, obtain the three-dimensional coordinates of the remaining reflective markers in the camera coordinate system.

[0069] As an example, for the L-shaped calibration rod shown as Figure 3 , the first number of reflective markers with the most collinear are the three reflective markers numbered 0, 1, and 2.

[0070] Specifically, when calculating the three-dimensional coordinates of the first number of reflective markers with the most collinear in the camera coordinate system, first set Figure 3 The three-dimensional coordinates of the four reflective markers in are respectively represented as (x c0 , y c0 , z c0 ), (x c1 , y c1 , z c1 ), (x c2 , yc2 , z c2 ), (x c3 , y c3 , z c3 ), and then, by using the collinear relationship between multiple reflective markers, a series of collinear equations can be formed, which are expressed as follows:

[0071]

[0072] Among them, L 01 represents the physical rod length between reflective markers 0 and 1, L 02 represents the physical rod length between reflective markers 0 and 2, and L 03 represents the physical rod length between reflective markers 0 and 3.

[0073] The normalized plane coordinates of the four reflective markers in the camera coordinate system are respectively represented as (x n0 , y n0 , 1), (x n1 , y n1 , 1), (x n2 , y n2 , 1), (x n3 , y n3 , 1). By using the optical center ray passing through the camera normalized plane coordinate points and the camera coordinate points, a series of ray equations can be obtained, which are expressed as follows:

[0074]

[0075] Based on formulas (1) and (2), a three-dimensional coordinate solving matrix A of the camera coordinate system is constructed. By performing singular value decomposition on matrix A, the three-dimensional coordinates of the collinear reflective markers 0, 1, and 2 in the camera coordinate system can be obtained, that is, (x c0 , y c0 , z c0 ), (x c1 , y c1 , z c1 ), (x c2 , y c2 , z c2 ) are solved. Furthermore, by using the cosine theorem and combining with the physical length L 03 , the three-dimensional coordinates (x c3 , y c3 , z c3 ) of the reflective marker 3 in the camera coordinate system can be calculated. Thus, the three-dimensional coordinates of all the reflective markers of the L-shaped calibration rod are obtained.

[0076] In some embodiments, after calculating the three-dimensional coordinates of multiple reflective markers on the cross marker linkage in the camera coordinate system based on the multi-frame images of the cross marker linkage in the step, the method further includes:

[0077] Processing the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system by using an optimization algorithm to obtain the processed three-dimensional coordinates;

[0078] Specifically, there are certain errors in the obtained three-dimensional coordinates of the multiple reflective markers in the camera coordinate system. Performing non-linear optimization of the three-dimensional points in the camera coordinate system on the three-dimensional coordinates can further improve the accuracy of the three-dimensional coordinates. The optimization algorithm used in this embodiment is the Levenberg-Marquardt algorithm. The input parameters are the normalized plane coordinates of the reflective markers in the camera coordinate system, the three-dimensional coordinates in the reference coordinate system, and the three-dimensional coordinates of the reflective markers calculated during the iterative process in the camera coordinate system. The optimization objective function is shown in formulas (3), (4), and (5):

[0079]

[0080] Among them, f1 represents the residual function composed of the physical distances between the reflective markers, and L i represents the actual physical distance between the reflective markers, such as the above-mentioned L 01 , L 02 , L 03 , and l i represents the physical distance between the reflective markers calculated in real time during the optimization iteration process.

[0081]

[0082] Among them, f2 represents the collineation equation residual sum and ray equation residual function composed of formulas (1) and (2), and Collineation i represents the calculated value of each equation composed of formula (1), and Ray j represents the calculated value of each equation composed of formula (2).

[0083] F(P c , P n , P w ) = f1(P c , P n , P w ) + f2(P c , P n , P w ) (5)

[0084] Among them, F(P c , P n , Pw ) represents the non - linear optimization function of the three - dimensional coordinates in the camera coordinate system for the final solution.

[0085] At this time, calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the three - dimensional coordinates of the multiple reflective markers in the camera coordinate system includes:

[0086] Calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the processed three - dimensional coordinates of the multiple reflective markers in the camera coordinate system.

[0087] In some embodiments, in order to obtain the rotation matrix from the reference coordinate system to the camera coordinate system based on the three - dimensional coordinates of the multiple reflective markers in the camera coordinate system, the embodiments of the present application adopt the following process. Specifically, as Figure 4 shown, step S103 includes the following steps:

[0088] S401, based on the three - dimensional coordinates of the multiple reflective markers in the camera coordinate system, translate the origin of the reference coordinate system to the origin of the camera coordinate system to obtain a new coordinate system;

[0089] S402, based on the new coordinate system, calculate the unit direction vectors of the three coordinate axes of the new coordinate system;

[0090] S403, based on the unit direction vectors of the three coordinate axes of the new coordinate system and the unit direction vectors of the respective reference coordinate axes of the camera coordinate system, calculate the direction cosine vectors of the three coordinate axes of the new coordinate system;

[0091] S404, according to the direction cosine vectors of the three coordinate axes of the new coordinate system, obtain the rotation matrix from the reference coordinate system to the camera coordinate system.

[0092] Specifically, based on the three - dimensional coordinates of the multiple reflective markers in the camera coordinate system, translate the three - dimensional coordinates as a whole to the origin of the camera coordinate system. The translation vector is the intersection point coordinate of the L - shaped calibration rod. After translating the intersection point coordinate of the L - shaped calibration rod to the origin of the camera coordinate system, based on the special position relationship of the L - shaped calibration rod, that is, the straight line passing through points 0, 1, and 2 on the cross - bar is perpendicular to the straight line passing through points 0 and 3. In this way, taking Line 012 and Line 03 as two new coordinate axes, the directions represented by the new coordinate axes are the same as the orientations of the coordinate axes of the reference coordinate system. For example, if Line 012 and Line 03 in the reference coordinate system coincide with the x - axis and y - axis respectively, then Line 012 and Line 03They also coincide with the x-axis and y-axis respectively. The newly established coordinate system can be expressed as the coordinate system after the reference coordinate system is translated to the origin of the camera coordinate system. The reference coordinate system refers to a unified world coordinate in the multi-camera system, that is, the coordinate system defined by the position of the cross marker link in the multi-camera system placement. It is the coordinate system that all cameras ultimately need to be unified to. Among them, the axes are composed of the positions of the reflective markers on the cross marker link in the actual environment, regarded as the reference coordinate axes. The external camera parameters, that is, the camera pose, refer to the rotation matrix and translation vector calculated for each camera to this reference coordinate system after the positions of the reflective calibration objects and the axes of the cross calibration objects are determined. The rotation matrix and translation vector of each camera to this reference coordinate system are different, and it also refers to the pose relationship between the camera coordinate system of each camera and this reference coordinate system. The schematic diagram of the relationship among the reference coordinate system, the camera coordinate system, and the newly established coordinate system is as Figure 5 shown. Through the newly established coordinate system, calculate the unit direction vectors of the three coordinate axes, and jointly solve them with the unit direction vectors of the reference coordinate axes of the camera coordinate system to calculate the direction cosine vectors of the three coordinate axes. The matrix composed of these direction cosine vectors is the rotation matrix from the reference coordinate system to the camera coordinate system.

[0093] In some embodiments, before the step of processing the rotation matrix and translation vector by using the optimization algorithm, the method further includes:

[0094] Orthogonalize the rotation matrix to obtain the initial value of the rotation matrix from the reference coordinate system to the camera coordinate system.

[0095] In some embodiments, there may be errors in the calculation process. For the direction cosine matrix obtained above, that is, the rotation matrix may not be an orthogonal matrix. Therefore, it needs to be orthogonalized. After orthogonal processing, the initial value of the rotation matrix from the reference coordinate system to the camera coordinate system can be obtained.

[0096] In some embodiments, the step of processing the rotation matrix and translation vector by using the optimization algorithm to obtain the processed rotation matrix and translation vector of each camera to the reference coordinate system includes:

[0097] Obtain the initial internal parameters of each camera and the pixel coordinates of the reflective markers;

[0098] Based on the initial internal parameters of each camera, the rotation matrix and translation vector of each camera to the reference coordinate system, the pixel coordinates of the reflective markers, and the three-dimensional coordinates in the camera coordinate system, obtain the processed initial internal parameters of each camera and the rotation matrix and translation vector of each camera to the reference coordinate system through the optimization algorithm.

[0099] Specifically, in order to obtain more accurate internal parameters of each camera and the attitude relationship between each camera and the reference coordinate system, the present application uses the Levenberg-Marquardt algorithm to jointly and nonlinearly optimize the internal and external parameters of the camera. In the optimization algorithm, the initial values of the internal parameters of each camera, the rotation matrix and translation vector from each camera to the reference coordinate system, as well as the pixel coordinates of the reflective markers and their three-dimensional coordinates in the camera coordinate system are input. The optimization objective function is expressed as follows:

[0100]

[0101] Among them, f union is the normal joint optimization function without exception handling. i represents the i-th camera combination, j represents the three-dimensional coordinates of the j-th current marker point, k represents the k-th pixel coordinate in the camera combination, q represents the q-th epipolar distance, BaseCoord 3dij and CalCoord 3dij respectively represent the reference three-dimensional coordinates and the calculated three-dimensional coordinates in the camera coordinate system. BasePixel 2dik and CalPixel 2dik respectively represent the reference pixel coordinates and the calculated pixel coordinates. D iq represents the distance from a point to an epipolar line.

[0102] In summary, the embodiments of the present application have the following beneficial effects:

[0103] For the external parameter calibration method of the multi-camera system described in the present application, in order to obtain the three-dimensional coordinates of the reflective markers on the cross-marker link, it is necessary to first obtain multiple frames of images collected by multiple cameras synchronously for the cross-marker link. A plurality of reflective markers are provided on the cross-marker link. Then, based on the multiple frames of images of the cross-marker link, the three-dimensional coordinates of the multiple reflective markers on the cross-marker link in the camera coordinate system are calculated, and according to the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system, the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system are calculated. Finally, in order to obtain a more accurate and robust attitude relationship between each camera and the reference coordinate system, an optimization algorithm is used to process the rotation matrix and translation vector to obtain the processed rotation matrix and translation vector from each camera to the reference coordinate system; for the external parameter calibration method described in the present application, by calculating the three-dimensional coordinates of the reflective markers on the cross-marker link, a new coordinate system is established according to the three-dimensional coordinates and the rotation matrix is calculated, and finally the rotation matrix and translation vector are optimized by an optimization algorithm, so that the attitude relationship between each camera and the reference coordinate system obtained is more accurate, that is, the accuracy and robustness of the external parameter calibration of each camera are improved.

[0104] Based on the same inventive concept, an external parameter calibration device for a multi-camera system corresponding to the external parameter calibration method of the multi-camera system in the first embodiment is also provided in the embodiments of the present application. Since the principle of solving problems by the device in the embodiments of the present application is similar to the above external parameter calibration method of the multi-camera system, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0105] As Figure 6 shown, Figure 6 is a schematic structural diagram of an external parameter calibration device for a multi-camera system provided by the present application. The external parameter calibration device for the multi-camera system is applied to the external parameter calibration of at least two cameras, and includes:

[0106] An acquisition module 601, configured to acquire multiple frames of images obtained by synchronously collecting cross-marker linkages by multiple cameras; a plurality of reflective markers are arranged on the cross-marker linkages;

[0107] A first calculation module 602, configured to calculate three-dimensional coordinates of a plurality of reflective markers on the cross-marker linkages in a camera coordinate system based on the multiple frames of images of the cross-marker linkages;

[0108] A second calculation module 603, configured to calculate a rotation matrix and a translation vector from a reference coordinate system to the camera coordinate system according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system;

[0109] A processing module 604, configured to process the rotation matrix and the translation vector by using an optimization algorithm to obtain the rotation matrix and the translation vector of each camera to the reference coordinate system after processing.

[0110] Those skilled in the art should understand that Figure 6 the implementation functions of the modules in the external parameter calibration device for the multi-camera system shown can be understood with reference to the relevant descriptions of the above external parameter calibration method of the multi-camera system. Figure 6 The functions of the units in the external parameter calibration device for the multi-camera system shown can be implemented by a program running on a processor or can be implemented by specific logic circuits.

[0111] In a possible implementation manner, the first calculation module 602 includes:

[0112] An analysis unit, configured to analyze the multiple frames of images of the cross-marker linkages to determine three-dimensional coordinates of the first number of reflective markers with the most collinear in a line segment formed by the plurality of reflective markers in the camera coordinate system;

[0113] A first determination unit, configured to obtain three-dimensional coordinates of the remaining reflective markers in the camera coordinate system based on the three-dimensional coordinates of the first number of reflective markers in the camera coordinate system.

[0114] In a possible implementation, after the multi-camera system external parameter calibration device calculates the three-dimensional coordinates of multiple reflective markers on the cross marker link in the camera coordinate system based on multiple frames of images of the cross marker link, it is further used for:

[0115] Using an optimization algorithm to process the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system to obtain the processed three-dimensional coordinates.

[0116] In this embodiment, calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system includes:

[0117] Calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system based on the processed three-dimensional coordinates of the multiple reflective markers in the camera coordinate system.

[0118] In a possible implementation, the second calculation module 603 includes:

[0119] A new construction unit for translating the origin of the reference coordinate system to the origin of the camera coordinate system based on the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system to obtain a new construction coordinate system;

[0120] A first calculation unit for calculating the unit direction vectors of the three coordinate axes of the new construction coordinate system based on the new construction coordinate system;

[0121] A second calculation unit for calculating the direction cosine vectors of the three coordinate axes of the new construction coordinate system based on the unit direction vectors of the three coordinate axes of the new construction coordinate system and the unit direction vectors of the respective reference coordinate axes of the camera coordinate system;

[0122] A second determination unit for obtaining the rotation matrix from the reference coordinate system to the camera coordinate system according to the direction cosine vectors of the three coordinate axes of the new construction coordinate system.

[0123] In a possible implementation, before the multi-camera system external parameter calibration device uses an optimization algorithm to process the rotation matrix and translation vector, it is further used for:

[0124] Orthogonalizing the rotation matrix to obtain an initial value of the rotation matrix from the reference coordinate system to the camera coordinate system.

[0125] In a possible implementation, the processing module 604 is used for:

[0126] Obtaining the initial internal parameters of each camera and the pixel coordinates of the reflective marker;

[0127] Based on the initial intrinsic parameters of each camera, the rotation matrices and translation vectors from each camera to the reference coordinate system, as well as the pixel coordinates and three-dimensional coordinates in the camera coordinate system of the retroreflective markers, the optimized algorithm is used to obtain the processed initial intrinsic parameters of each camera and the rotation matrices and translation vectors from each camera to the reference coordinate system.

[0128] In a possible implementation manner, there are no less than four retroreflective markers provided on the cross marker link, and the distances between adjacent retroreflective markers are all different;

[0129] The line segments formed by sequentially connecting all the retroreflective markers are two mutually perpendicular line segments.

[0130] For the above multi-camera system external parameter calibration device, in order to obtain the three-dimensional coordinates of the retroreflective markers on the cross marker link, first, the acquisition module 601 is used to acquire multiple frames of images obtained by multiple cameras synchronously collecting the cross marker link. There are multiple retroreflective markers provided on the cross marker link. Then, based on the multiple frames of images of the cross marker link, the first calculation module 602 calculates the three-dimensional coordinates of the multiple retroreflective markers on the cross marker link in the camera coordinate system, and according to the three-dimensional coordinates of the multiple retroreflective markers in the camera coordinate system, the second calculation module 603 calculates the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system. Finally, in order to obtain a more accurate and robust attitude relationship from each camera to the reference coordinate system, the processing module 604 uses an optimization algorithm to process the rotation matrix and translation vector to obtain the processed rotation matrix and translation vector from each camera to the reference coordinate system; for the usage method of the external parameter calibration device of the present application, by calculating the three-dimensional coordinates of the retroreflective markers on the cross marker link, a new coordinate system is established according to the three-dimensional coordinates and the rotation matrix is calculated. Finally, the rotation matrix and translation vector are optimized by an optimization algorithm, so that the obtained attitude relationship from each camera to the reference coordinate system is more accurate, that is, the accuracy and robustness of the external parameter calibration of each camera are improved.

[0131] Corresponding to Figure 1 in the multi-camera system external parameter calibration method, the embodiment of the present application further provides a computer device 700, as Figure 7 shown. This device includes a memory 701, a processor 702, and a computer program stored on the memory 701 and executable on the processor 702. Among them, when the above processor 702 executes the above computer program, the above multi-camera system external parameter calibration method is implemented.

[0132] Specifically, the above-mentioned memory 701 and processor 702 can be general-purpose memory and processor, which are not specifically limited here. When the processor 702 runs the computer program stored in the memory 701, it can execute the above-mentioned multi-camera system external parameter calibration method, solving the problem of low accuracy and robustness of the calibration results of the attitude calibration from multiple cameras to the reference coordinate system in the prior art.

[0133] Corresponding to Figure 1 For the multi-camera system external parameter calibration method in [[ ]], an embodiment of the present application also provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the steps of the above-mentioned multi-camera system external parameter calibration method.

[0134] Specifically, the storage medium can be a general-purpose storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned multi-camera system external parameter calibration method, solving the problem of low accuracy and robustness of the calibration results of the attitude calibration from multiple cameras to the reference coordinate system in the prior art.

[0135] For the above-mentioned computer-readable storage medium, in order to obtain the three-dimensional coordinates of the reflective markers on the cross-marker link, it is necessary to first obtain multiple frames of images obtained by multiple cameras synchronously collecting the cross-marker link. A plurality of reflective markers are arranged on the cross-marker link. Then, based on the multiple frames of images of the cross-marker link, calculate the three-dimensional coordinates of the multiple reflective markers on the cross-marker link in the camera coordinate system, and calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system according to the three-dimensional coordinates of the multiple reflective markers in the camera coordinate system. Finally, in order to obtain a more accurate and robust attitude relationship of each camera to the reference coordinate system, an optimization algorithm is used to process the rotation matrix and translation vector to obtain the processed rotation matrix and translation vector of each camera to the reference coordinate system; the usage method of the computer-readable storage medium of the present application, by calculating the three-dimensional coordinates of the reflective markers on the cross-marker link, and creating a new coordinate system according to the three-dimensional coordinates and calculating the rotation matrix, and finally optimizing the rotation matrix and translation vector through an optimization algorithm, makes the obtained attitude relationship of each camera to the reference coordinate system more accurate, that is, improves the accuracy and robustness of the external parameter calibration of each camera.

[0136] In the embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in the embodiments provided in the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0139] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or this part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0140] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0141] It should be noted that the term "including" used in the embodiments of the present application is used to indicate the existence of the features stated thereafter, but does not exclude adding other features.

[0142] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application and are not intended to limit this application.

[0143] Finally, it should be noted that the above-described embodiments are only specific embodiments of this application, which are used to illustrate the technical solutions of this application and are not intended to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art can still modify the technical solutions described in the foregoing embodiments or easily conceive of changes within the technical scope disclosed in this application, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All of them should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for calibrating the external parameters of a multi-camera system, which is applied to the calibration of the external parameters of at least two cameras, It is characterized in that comprising the following steps: Obtain multiple frames of images collected synchronously by multiple cameras of a cross-marker linkage; a plurality of reflective markers are arranged on the cross-marker linkage; Based on the multiple frames of images of the cross-marker linkage, calculate the three-dimensional coordinates of the plurality of reflective markers on the cross-marker linkage in the camera coordinate system; According to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system, calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system; Use an optimization algorithm to process the rotation matrix and translation vector to obtain the rotation matrix and translation vector from each camera to the reference coordinate system after processing.

2. The method for calibrating the external parameters of a multi-camera system according to claim 1, wherein the calculating the three-dimensional coordinates of the plurality of reflective markers on the cross-marker linkage in the camera coordinate system based on the multiple frames of images of the cross-marker linkage includes: Analyze the multiple frames of images of the cross-marker linkage, and determine the three-dimensional coordinates of the first number of reflective markers with the most collinear ones in the line segment formed by the plurality of reflective markers in the camera coordinate system; Based on the three-dimensional coordinates of the first number of reflective markers in the camera coordinate system, obtain the three-dimensional coordinates of the remaining reflective markers in the camera coordinate system.

3. The method for calibrating the external parameters of a multi-camera system according to claim 1, wherein After the step of calculating the three-dimensional coordinates of the plurality of reflective markers on the cross-marker linkage in the camera coordinate system based on the multiple frames of images of the cross-marker linkage, the method further includes: Use an optimization algorithm to process the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system to obtain the processed three-dimensional coordinates; The calculating the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system includes: Calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system according to the processed three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system.

4. The external parameter calibration method for a multi-camera system according to claim 1, wherein, The calculating the rotation matrix from the reference coordinate system to the camera coordinate system according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system includes: Based on the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system, translate the origin of the reference coordinate system to the origin of the camera coordinate system to obtain a new coordinate system; Based on the new coordinate system, calculate the unit direction vectors of the three coordinate axes of the new coordinate system; Based on the unit direction vectors of the three coordinate axes of the new coordinate system and the unit direction vectors of the respective reference coordinate axes of the camera coordinate system, calculate the direction cosine vectors of the three coordinate axes of the new coordinate system; According to the direction cosine vectors of the three coordinate axes of the new coordinate system, obtain the rotation matrix from the reference coordinate system to the camera coordinate system.

5. The method for calibrating the external parameters of a multi-camera system according to claim 1, wherein, Before the step of using an optimization algorithm to process the rotation matrix and translation vector, the method further includes: Orthogonalize the rotation matrix to obtain the initial value of the rotation matrix from the reference coordinate system to the camera coordinate system.

6. The multi-camera system extrinsic parameter calibration method according to claim 1, characterized in that, Processing the rotation matrix and the translation vector by using an optimization algorithm to obtain the processed rotation matrix and translation vector of each camera to the reference coordinate system, including: Obtaining the initial internal parameters of each camera and the pixel coordinates of the reflective markers; Based on the initial internal parameters of each camera, the rotation matrix and translation vector of each camera to the reference coordinate system, the pixel coordinates of the reflective markers, and the three-dimensional coordinates in the camera coordinate system, obtaining the processed initial internal parameters of each camera and the rotation matrix and translation vector of each camera to the reference coordinate system through an optimization algorithm.

7. The method for calibrating the external parameters of the multi-camera system according to claim 1, wherein There are no less than four reflective markers provided on the cross marker link, and the distances between adjacent reflective markers are all different; The line segments formed by connecting all the reflective markers in sequence are two mutually perpendicular line segments.

8. An external parameter calibration device for a multi-camera system, which is applied to the external parameter calibration of at least two cameras, and is characterized in that, Including: An acquisition module, configured to acquire multiple frames of images obtained by synchronously collecting a cross marker link by multiple cameras; a plurality of reflective markers are provided on the cross marker link; A first calculation module, configured to calculate the three-dimensional coordinates of the plurality of reflective markers on the cross marker link in the camera coordinate system based on the multiple frames of images of the cross marker link; A second calculation module, configured to calculate the rotation matrix and translation vector from the reference coordinate system to the camera coordinate system according to the three-dimensional coordinates of the plurality of reflective markers in the camera coordinate system; A processing module, configured to process the rotation matrix and translation vector by using an optimization algorithm to obtain the processed rotation matrix and translation vector of each camera to the reference coordinate system.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method described in any one of the above claims 1-7 are implemented.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is run by the processor, the steps of the method described in any one of the above claims 1-7 are executed.

Images