Cooperative calibration method for multiple cameras with different resolutions

Abstract

The invention relates to a cooperative calibration method for multiple cameras with different resolutions, and the method comprises the steps: controlling all cameras to move in a unified manner, shooting images in the movement process, and enabling the scene coincidence rate of any adjacent frame images to meet a preset condition; point cloud reconstruction is carried out on all images of each camera, an internal reference matrix of each camera under an independent coordinate system is obtained, a three-dimensional point cloud model is established for each camera, one camera is selected as a reference camera, and the other cameras are used as to-be-coordinated cameras; registering the three-dimensional point cloud model of any camera to be coordinated with the three-dimensional point cloud model of the reference camera, and selecting a plurality of reference points from the registered three-dimensional point cloud models; and determining a position relation matrix of the current to-be-coordinated camera and the reference camera according to the plurality of reference points, and determining a coordinated external parameter matrix of the current to-be-coordinated camera according to the position relation matrix and the internal parameter matrix of the reference camera. The method can be suitable for cameras with different depths of field and different resolutions.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera calibration, and particularly relates to a collaborative calibration method for multi-cameras with different resolutions. Background Art

[0002] Multi-camera systems are widely used in various task requirements such as target tracking, panoramic imaging, and three-dimensional scene reconstruction. The accurate calibration of these cameras is a prerequisite for the application of multi-camera systems in panoramic imaging, three-dimensional reconstruction, target tracking, and size measurement. In particular, in the field of computer vision, it is often necessary to locate, estimate the size, and measure targets of various sizes and at various distances in different scenarios, and multiple camera sensors must be used to obtain the three-dimensional information of the target object. Moreover, in applications with large field-of-view requirements, the number of cameras involved is significantly increased, and the focal lengths, depths of field, and resolutions of each camera may be different. To maximize the field of view, the overlapping area of the cameras may be small, thus posing a great challenge to calibration. The commonly used Zhang Zhengyou calibration method in the field is limited to the near field, and its applicable conditions are very limited. Summary of the Invention

[0003] The present invention provides a collaborative calibration method for multi-cameras with different resolutions, which does not rely on the commonly used calibration targets and has a scalable multi-camera calibration method, and can be applied to cameras with different depths of field and different resolutions to solve the technical problems mentioned in the background art.

[0004] The technical solution of the present invention is as follows: A collaborative calibration method for multi-cameras with different resolutions, comprising:

[0005] S10: Control all cameras to move uniformly, capture images during the movement, and the scene overlap rate of any adjacent frame images needs to meet a preset condition;

[0006] S20: Perform point cloud reconstruction on all images of each camera to obtain the internal parameter matrix in the independent coordinate system of each camera, establish a three-dimensional point cloud model for each camera, select one camera as the reference camera, and the remaining cameras as the cameras to be collaborated;

[0007] S30: Register the three-dimensional point cloud model of any camera to be collaborated with the three-dimensional point cloud model of the reference camera, and select multiple reference points from the registered three-dimensional point cloud model;

[0008] S40: Determine the position relationship matrix between the current camera to be collaborated and the reference camera according to the multiple reference points, and determine the collaborative external parameter matrix of the current camera to be collaborated according to the position relationship matrix and the internal parameter matrix of the reference camera;

[0009] S50: Repeat S30 - S40 to determine the collaborative external parameter matrices of all cameras to be collaborated.

[0010] Further, in the step S10, the preset condition is that the scene coincidence rate of any adjacent frame images needs to be greater than 20%.

[0011] Further, the step S20 includes:

[0012] Define the camera imaging model:

[0013]

[0014] Written in matrix-vector form as: u n,k,j = A n [R n,k | t n,k x j , representing the image point corresponding to the spatial point, A n is the internal parameter matrix of the camera to be coordinated, is the external parameter matrix of the camera to be coordinated when collecting the k-th image, is the homogeneous coordinate of a point, that is, the column vector (x, y, z, 1) T , 0 is the vector with a value of 0.

[0015] Further, the step S30 includes:

[0016] Define the relationship between a pair of matching reference points in the coordinate systems of the n-th camera and the n'-th camera after registration as:

[0017]

[0018] where, x n,k is the coordinate vector of a reference point in the coordinate system of the n-th camera, x n',k is the coordinate vector of the corresponding reference point in the coordinate system of the n'-th camera, R n→n' is the rotation matrix from the coordinate system of the n-th camera to the coordinate system of the n'-th camera, t n→n' is the translation matrix from the coordinate system of the n-th camera to the coordinate system of the n'-th camera.

[0019] Further, the step S40 includes:

[0020] The currently coordinated camera calculates the position relationship matrix with the reference camera through the first formula;

[0021] Calculate the coordinated external parameter matrix through the position relationship matrix between the currently coordinated camera and the reference camera and the internal parameter matrix of the reference camera.

[0022] Further, the first formula includes:

[0023]

[0024] Or:

[0025]

[0026] Or:

[0027]

[0028] Wherein, and are the homogeneous coordinates corresponding to the registration reference points under the reference camera and the current camera to be coordinated respectively, u n is the image point coordinate of the registration reference point on the current camera to be coordinated, λ is an adjustable hyperparameter, A n is the internal parameter matrix of the current camera to be coordinated, is the external parameter matrix of the current camera to be coordinated when collecting the k-th image.

[0029] Furthermore, the step S40 includes:

[0030] The current camera to be coordinated calculates the position relationship matrix with the previous camera to be coordinated through the third formula;

[0031] Calculate the position relationship matrix between the current camera to be coordinated and the reference camera through the position relationship matrix between the current camera to be coordinated and the previous camera to be coordinated and the external parameter matrix of the current camera to be coordinated;

[0032] Calculate the collaborative external parameter matrix through the position relationship matrix between the current camera to be coordinated and the reference camera and the internal parameter matrix of the reference camera.

[0033] Furthermore, the third formula includes:

[0034]

[0035] Or:

[0036]

[0037] Or:

[0038]

[0039] Wherein, and are the homogeneous coordinates corresponding to the registration reference points of the previous camera to be coordinated and the current camera to be coordinated respectively, u n is the image point coordinate of the registration reference point on the current camera to be coordinated, λ is an adjustable hyperparameter, A n is the internal parameter matrix of the current camera to be coordinated, is the external parameter matrix of the current camera to be coordinated.

[0040] Further, the formula for calculating the position relationship matrix between the current camera to be coordinated and the reference camera through the position relationship matrix between the current camera to be coordinated and the previous camera to be coordinated and the external parameter matrix of the current camera to be coordinated is as follows:

[0041]

[0042] Further, the coordinated external parameter matrix is:

[0043]

[0044] where is the external parameter matrix of the reference camera.

[0045] Advantages of the present invention: The method of the present invention does not rely on the conventionally used calibration targets. Especially in the situation where it is difficult to obtain targets under the application requirements of large scenes, a practical and easy-to-implement method is provided. The method of the present invention is convenient for expansion and can be directly extended to multiple cameras, without limiting the number of cameras and the camera layout, and can be applied to cameras with different depths of field and different resolutions. In addition, the present invention is based on the registered reference points rather than physical reference points, which is convenient for introducing cutting-edge technologies such as deep learning and extending to the mutual calibration and information fusion of different modality cameras such as infrared cameras. Description of the Drawings

[0046] Figure 1 is the flowchart of the present invention. Detailed Embodiment

[0047] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0048] In the technical solution of the present invention, Figure 1 is a flowchart provided according to the specific steps of a method for cooperative calibration of multiple cameras with different resolutions according to the present invention. As Figure 1 shown, it includes:

[0049] S10: Control all cameras to move uniformly, and capture images during the movement. The scene coincidence rate of any adjacent frame images needs to meet the preset conditions.

[0050] Place the multi-camera imaging system on an orbit for controllable movement, and collect a series of images during the movement. Denote as g n,k, where k is the index of the image sequence. Each camera of the imaging system ensures that the images captured during its movement meet the resolution requirements, and the adjacent frame images contain a sufficiently large scene overlap area.

[0051] Specifically, the scene overlap rate of any adjacent frame images needs to be greater than 20%.

[0052] S20: Perform point cloud reconstruction on all the images of each camera, obtain the internal parameter matrix in the independent coordinate system of each camera, establish a three-dimensional point cloud model for each camera, select one camera as the reference camera, and the remaining cameras as the cameras to be coordinated. If the calibration fails (i.e., the reconstructed point cloud is too sparse), return to S10 to re-collect, and the scene can be changed for re-collection.

[0053] Define the camera imaging model:

[0054]

[0055] Written in matrix-vector form as: u n,k,j = A n [R n,k |t n,k x j , representing the image point corresponding to the spatial point, A n is the internal parameter matrix of the camera to be coordinated, is the external parameter matrix of the camera to be coordinated when capturing the k-th image, is a point homogeneous coordinate, that is, the column vector (x, y, z, 1) in the above formula T , and 0 is a vector with a value of 0.

[0056] For three-dimensional initial point cloud reconstruction, existing methods in the field such as COLMAP or SfM can be used, so it will not be elaborated here. Obtain the internal parameter matrix A n and the external parameter matrix in the independent coordinate system of this camera. Use the 3D Gaussian splatting (3DGS) method for three-dimensional scene reconstruction.

[0057] S30: Register the three-dimensional point cloud model of any camera to be coordinated with the three-dimensional point cloud model of the reference camera, and select multiple reference points from the registered three-dimensional point cloud model. Specifically, it should be noted that the number of reference points is greater than or equal to 6. It should be noted that if there are insufficient reference points corresponding to some cameras, return to S10 to re-collect data.

[0058] Define the relationship between a pair of matching reference points in the coordinate systems of the n-th camera and the n'-th camera after registration as:

[0059]

[0060] where x n,k is the coordinate vector of a reference point in the camera coordinate system of the nth camera, and x n',k is the coordinate vector of the corresponding reference point in the camera coordinate system of the n'th camera, and R n→n' is the rotation matrix from the camera coordinate system of the nth camera to the camera coordinate system of the n'th camera, and t n→n' is the translation matrix from the camera coordinate system of the nth camera to the camera coordinate system of the n'th camera.

[0061] S40: Determine the position relationship matrix between the currently to-be-collaborated camera and the reference camera according to multiple said reference points, and determine the collaborative external parameter matrix of the currently to-be-collaborated camera according to the position relationship matrix and the internal parameter matrix of the reference camera.

[0062] Specific method 1:

[0063] The currently to-be-collaborated camera calculates the position relationship matrix with the reference camera through the first formula;

[0064] Calculate the collaborative external parameter matrix through the position relationship matrix between the currently to-be-collaborated camera and the reference camera and the internal parameter matrix of the reference camera.

[0065] Among them, the first formula includes:

[0066]

[0067] Or:

[0068]

[0069] Or:

[0070]

[0071] Among them, where and are the homogeneous coordinates corresponding to the registered reference points under the reference camera and the currently to-be-collaborated camera respectively, u n is the image point coordinate of the registered reference point in the currently to-be-collaborated camera, λ is an adjustable hyperparameter, and A n is the internal parameter matrix of the currently to-be-collaborated camera, is the external parameter matrix of the currently to-be-collaborated camera when collecting the kth image.

[0072] The collaborative external parameter matrix is:

[0073]

[0074] Among them, is the external parameter matrix of the reference camera.

[0075] Specific method 2:

[0076] The current camera to be coordinated calculates the position relationship matrix with the previous camera to be coordinated through the third formula;

[0077] Calculate the position relationship matrix between the current camera to be coordinated and the reference camera through the position relationship matrix between the current camera to be coordinated and the previous camera to be coordinated and the external parameter matrix of the current camera to be coordinated;

[0078] Calculate the collaborative external parameter matrix through the position relationship matrix between the current camera to be coordinated and the reference camera and the internal parameter matrix of the reference camera.

[0079] Among them, the third formula includes:

[0080]

[0081] Or:

[0082]

[0083] Or:

[0084]

[0085] Among them, and are the homogeneous coordinates of the registration reference points corresponding to the previous camera to be coordinated and the current camera to be coordinated respectively, u n is the image point coordinate of the registration reference point in the current camera to be coordinated, λ is an adjustable hyperparameter, A n is the internal parameter matrix of the current camera to be coordinated, is the external parameter matrix of the current camera to be coordinated.

[0086] The formula for calculating the position relationship matrix between the current camera to be coordinated and the reference camera through the position relationship matrix between the current camera to be coordinated and the previous camera to be coordinated and the external parameter matrix of the current camera to be coordinated is:

[0087]

[0088] The collaborative external parameter matrix is:

[0089]

[0090] Among them, is the external parameter matrix of the reference camera.

[0091] S50: Repeat S30 - S40 to determine the collaborative external parameter matrices of all cameras to be coordinated.

[0092] Finally, it should be noted that the above specific implementation manners are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the examples, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and all of them should be covered by the scope of the claims of the present invention.

Claims

1. A collaborative calibration method for multiple cameras with different resolutions, characterized in that: include: S10: Control all cameras to move uniformly, capture images during the movement, and the scene overlap rate of any adjacent frame images needs to meet preset conditions; S20: reconstruct point clouds for all images of each camera, obtain the intrinsic parameter matrix of each camera in an independent coordinate system, establish a three-dimensional point cloud model for each camera, select one of the cameras as a reference camera, and the remaining cameras as cameras to be coordinated; S30: registering the 3D point cloud model of any camera to be coordinated with the 3D point cloud model of the reference camera, and selecting a plurality of reference points from the registered 3D point cloud models; S40: determining a position relationship matrix between the current camera to be coordinated and the reference camera according to the plurality of reference points, and determining a coordination extrinsic parameter matrix of the current camera to be coordinated according to the position relationship matrix and the intrinsic parameter matrix of the reference camera; S50: Repeat S30-S40 to determine the collaborative extrinsic parameter matrices of all the cameras to be coordinated.

2. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 1, characterized in that: In the step S10, a preset condition is that the scene overlap rate of any adjacent frame images needs to be greater than 20%.

3. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 1, characterized in that: The step S20 comprises: Define the camera imaging model: Among them, A n is the intrinsic parameter matrix of the camera to be coordinated, is the external parameter matrix of the camera to be coordinated when capturing the kth image, column vector (x, y, z, 1) T is the homogeneous coordinates of a point.

4. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 3, characterized in that: The step S30 comprises: The relationship between a pair of matching reference points in the coordinate system of the nth camera and the n'th camera after registration is defined as: Among them, x n,k is the coordinate vector of a reference point in the coordinate system of the nth camera, x n',k The coordinate vector of the corresponding reference point in the n'th camera coordinate system, R n→n' is the rotation matrix from the nth camera coordinate system to the n'th camera coordinate system, t n→n' is the translation matrix from the nth camera coordinate system to the n'th camera coordinate system.

5. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 1, characterized in that: The step S40 comprises: The current camera to be coordinated calculates the position relationship matrix with the reference camera through the first formula; The collaborative extrinsic parameter matrix is ​​calculated through the position relationship matrix between the current camera to be coordinated and the reference camera and the intrinsic parameter matrix of the reference camera.

6. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 5, characterized in that: The first formula includes: or: or: Among them, and The reference points for registration correspond to the homogeneous coordinates of the reference camera and the current camera to be coordinated, u n is the image point coordinate of the reference point to be registered in the current collaborative camera, λ is an adjustable hyperparameter, A n is the intrinsic parameter matrix of the current camera to be coordinated, is the external parameter matrix of the current cooperative camera when capturing the kth image.

7. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 1, characterized in that: The step S40 comprises: The current camera to be coordinated calculates the position relationship matrix with the previous camera to be coordinated by the third formula; Calculate the position relationship matrix between the current camera to be coordinated and the reference camera through the position relationship matrix between the current camera to be coordinated and the previous camera to be coordinated and the external parameter matrix of the current camera to be coordinated; The collaborative extrinsic parameter matrix is ​​calculated through the position relationship matrix between the current camera to be coordinated and the reference camera and the intrinsic parameter matrix of the reference camera.

8. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 7, characterized in that: The third formula includes: or: or: in, and The reference points for registration correspond to the coordinates of the previous camera to be coordinated and the current camera to be coordinated, u n is the image point coordinate of the reference point to be registered in the current collaborative camera, λ is an adjustable hyperparameter, A n is the intrinsic parameter matrix of the current camera to be coordinated, is the external parameter matrix of the current camera to be coordinated.

9. The collaborative calibration method for multiple cameras with different resolutions as claimed in claim 8, characterized in that: The formula for calculating the position relationship matrix between the current camera to be coordinated and the reference camera through the position relationship matrix between the current camera to be coordinated and the previous camera to be coordinated and the external parameter matrix of the current camera to be coordinated is:

10. The collaborative calibration method for multiple cameras with different resolutions according to any one of claims 5 to 9, characterized in that: The collaborative external parameter matrix is: in, is the extrinsic parameter matrix of the reference camera.

Images