Multi-view vision calibration method and system

Abstract

The present invention discloses a multi-camera calibration method and system, which solves the problem that the field of view of a multi-camera system cannot cover a two-dimensional calibration object. The method numbers the cameras 0 to N, and the calibration steps are as follows: First, camera 0 and camera 1 simultaneously observe the complete calibration plate, and the relative pose of the two cameras is calculated by extracting landmarks to obtain a relative pose transformation matrix. Secondly, camera 1 and camera 2 capture a complete calibration plate image, calculate the relative pose of camera 1 and camera 2, and obtain the relative pose matrix of the other camera. By calculating the rigid transformation relationship between the two pose matrices, the relative pose transformation relationship between camera 2 and camera 0 can be obtained. Similarly, the pose relationship of cameras 0-N can be obtained. The present invention can simplify the multi-camera calibration method and perform calibration when the cameras observe different calibration objects; the pose matrices of multiple cameras are calculated through relative poses and unified in a unified coordinate system.

Description

Technical Field

[0001] The present invention relates to the field of multi-viewing, and in particular to a multi-viewing calibration method and system. Background Art

[0002] With the continuous development of 3D measurement and multi-camera vision technologies, traditional single-camera measurement can no longer meet the requirements of industrial production, and multi-camera vision measurement has become a research hotspot. The first step in using multi-camera vision for 3D measurement is camera calibration. Only after camera calibration can the camera be linked to the world coordinate system, thus enabling 3D measurement.

[0003] Currently, most camera calibrations use two-dimensional calibration objects. For multi-camera vision systems, the two-dimensional calibration objects cannot sometimes be observed by all cameras. Usually, the camera-by-camera calibration method is adopted to calculate the intrinsic and extrinsic parameters of all cameras. As the number of cameras increases, the number of calibration images that need to be collected will increase exponentially, and the workload is too heavy. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-viewing calibration method and system.

[0005] The technical solution to achieve the purpose of the present invention is as follows: In a first aspect, the present invention provides a multi-viewing calibration method, comprising:

[0006] Build a multi-camera system, using a calibration plate with known landmark parameters, so that the field of view of at least two cameras can simultaneously cover the same calibration plate image;

[0007] The pixel coordinates of the landmarks are found through image morphology-related algorithms, the calibration plate pose is transformed, and different calibration images are collected to find the optimal solution. The internal and external parameters of the camera, including focal length, distortion coefficient, scale factor, and the rigid transformation matrix of the relative pose, are solved.

[0008] Using the solved rigid transformation matrix, the reference camera is set and the cameras are unified into the same coordinate system through calculation.

[0009] In one embodiment, the field of view of a single camera of the multi-camera system can cover the entire calibration plate.

[0010] In one embodiment, in a multi-camera system, adjacent cameras observe at least one calibration image simultaneously, ensuring that the relative positions of the adjacent cameras are calculated using the calibration image.

[0011] In one embodiment, the pixel coordinates of the marker points are found through an image morphology-related algorithm, the calibration plate posture is transformed, and different calibration images are collected to find the optimal solution. The intrinsic and extrinsic parameters of the camera, including focal length, distortion coefficient, scale factor, and the rigid transformation matrix of the relative posture, are solved as follows:

[0012] Step S21: extract the landmark points and calculate their pixel coordinates using the formula:

[0013]

[0014] Where (X c ,Y c ,Z c ) represents pixel coordinates, (X w ,Y w ,Z w ) represents the world coordinate, R is the 3x3 orthogonal rotation matrix, t is the three-dimensional translation vector, O T For the 0 vector matrix of corresponding size, calculate the values ​​of R and t;

[0015] Step S22: Capture multiple images of the calibration plate and calculate the optimal solution d(θ) for the camera parameters using the following formula:

[0016]

[0017] The coordinates of the center of the dot obtained by the calibration plate are M j , extract the center coordinates of the camera calibration image, the first camera is recorded as m j,k,1 , the second camera is denoted as m j,k,2 , the Nth camera is recorded as m j,k,l , and θ is used to represent the parameters of the two cameras, including the internal parameters and relative positions of the two cameras, where π i Represents the projection of a certain mark point on the calibration plate to the image plane coordinate system, θ i,l is the subset of camera parameters in camera l that affect this projection; π r Indicates the correction of a point in the image to the image plane coordinate system, θ r,l represents the subset of parameters in camera l that affect this correction; v j,k,l Indicates whether this point is valid; n0 is the number of calibration images; n m is the number of landmark points;

[0018] Step S23: Set any one of the cameras as the reference camera, that is, as the coordinate origin, and calculate the intrinsic and extrinsic parameters of the camera;

[0019] In one embodiment, the reference camera is set using the solved rigid transformation matrix, and the cameras are unified into the same coordinate system through calculation, specifically:

[0020] Step S31: The world coordinate point P of the calibration plate with the same posture observed by the adjacent cameras in step S22 w (x,y,z,1) T , then the transformation relationship between two adjacent cameras is:

[0021]

[0022] in and The world coordinate system of the calibration plate with the same pose observed by different cameras is obtained by solving the relevant parameters to obtain the transformation relationship of the relative pose, where R is the rotation matrix and t is the translation increment;

[0023] Step S32: Based on the relative pose relationships of different cameras calculated in S31, the pose relationships of all cameras in the system are further calculated.

[0024] In a second aspect, the present invention provides a multi-view visual calibration system, comprising:

[0025] Multi-camera system, cameras are numbered 0-N, using a calibration plate with known landmark parameters, and the field of view of at least two cameras can simultaneously cover a calibration plate image;

[0026] The calibration module uses image morphology-related algorithms to find the pixel coordinates of the landmark points, transform the calibration plate posture, collect different calibration images to find the optimal solution, and solve the internal and external parameters of the camera, including focal length, distortion coefficient, scale factor, and the rigid transformation matrix of the relative posture; using the solved rigid transformation matrix, set the reference camera and unify the cameras into the same coordinate system through calculation.

[0027] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0029] Compared with the prior art, the present invention has the following significant advantages:

[0030] (1) The present invention can simplify the multi-view calibration method and perform calibration when the camera observes different calibration objects; (2) the pose matrices of multiple cameras are calculated through relative poses and unified in a unified coordinate system; BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a calibration relationship diagram for the multi-camera system targeted by the present invention.

[0032] Figure 2 This is a flow chart of the multi-viewing calibration method of the present invention.

[0033] Figure 3 The calibration plate mentioned in the specific implementation manner of the present invention is a dot matrix coplanar circle calibration plate. DETAILED DESCRIPTION

[0034] The following will be combined with the Figures 1 and 2 , the method in this aspect is described in detail. The described method is part of a multi-camera system, not the entire system. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without inventive efforts are within the scope of protection of the present invention.

[0035] like Figure 1 This is a calibration diagram of a multi-camera system. Taking a multi-camera vision system consisting of five cameras as an example, the cameras are numbered 0-4, and the solid arrows indicate the directions of the camera's field of view.

[0036] S1, build a multi-camera system, the cameras are numbered 0-N, use a calibration plate with obvious marking points and known parameters, and at least two cameras' fields of view can cover a calibration plate image at the same time, ensuring that the cameras have Figure 1 The multi-camera system meets the following conditions:

[0037] 1) The calibration plate must have obvious marking points, and the position coordinates of the marking points are known. For example, Zhang's calibration plate or coplanar circle calibration plate. In this embodiment, the calibration plate is as follows Figure 3 shown.

[0038] 2) The field of view of a single camera in a multi-camera system must be able to cover the entire calibration plate, and the landmarks must be clearly visible in the field of view.

[0039] 3) After collecting the image, image preprocessing is required to ensure that the location of the landmarks can be extracted through image morphology methods in the subsequent process.

[0040] 4) For a multi-camera system, adjacent cameras observe at least one calibration image simultaneously to ensure that the relative poses of adjacent cameras can be calculated using the calibration image.

[0041] In the present invention, the multi-camera system is constructed with a "chain" relationship, where the chain relationship means that at least two adjacent cameras can observe the calibration plate at the same time. Figure 1 Cameras 0-2, three cameras simultaneously take a calibration picture, where the number of landmarks in the calibration picture is greater than or equal to 6, and there are obvious landmarks. The center position of the dot in the picture, the size of the calibration plate and the position of the dot are known, and its plane coordinates in the world coordinate system are (X w ,Y w ,1), the pixel coordinates of the captured image are (u,v,1).

[0042] S2 uses image morphology-related algorithms to find the pixel coordinates of the landmark points, transform the calibration plate posture, collect different calibration images to find the optimal solution, and solve the internal and external parameters of the camera, including focal length, distortion coefficient, scale factor, and rigid transformation matrix of relative posture.

[0043] In the present invention, Figure 1 Cameras 0-2 can simultaneously capture images of a certain pose of the calibration plate, that is, there is a complete calibration plate with clear markings in the image of camera 0-2. According to the relationship between the pixel coordinate system and the world coordinate system, there is a formula:

[0044]

[0045] in:

[0046]

[0047] r1, r2 are rotation matrices, T is the translation variable, Z c is the distance between the calibration plane and the camera, a x ,a y are the scale factors for the horizontal and vertical axes of the image, respectively. u0 and v0 are the coordinates of the origin of the image coordinate system in the pixel coordinate system. The parameters of K only include the focal length, principal point coordinates, etc., which are determined by the internal structure of the camera. Therefore, K is called the internal parameter matrix. H is the homography matrix formed by the product of the intrinsic and extrinsic parameters. Simultaneously, r1, r2, and T, the relative pose matrix of the camera, are calculated based on the relationship between the camera's internal and external parameters.

[0048] Assume that the camera parameters extracted from the image are θ=(f,k,s x ,s y ,c x ,c y ,α,β,γ,t x ,t y ,t z ), where f, k, s x ,s y ,c x ,c y are internal parameters, namely focal length, distortion coefficient, x-axis scaling factor, y-axis scaling factor, x-axis coordinate of the distortion center, and y-axis coordinate of the distortion center. x ,t y ,t z are external parameters, which are the rotation angle around the x-axis, the rotation angle around the y-axis, the rotation angle around the z-axis, the x-axis translation component, the y-axis translation component, and the z-axis translation component. Here, the projection of the world coordinate system to the image coordinate system is π i (M j ,θ i ), where Mj Represents the three-dimensional coordinates in the world coordinate system, let m j The coordinates of the center point of the solid circle of the calibration plate (marker point) in the image coordinate system. By calculating m j π calculated by projection i (M j ,θ i ) to determine the camera parameters.

[0049]

[0050] where n m is the number of marking points on the calibration plate, v i Indicates whether the marker is visible, 1 if visible, 0 otherwise. The number of parameters to be optimized in the above formula is too large, and if the initial value is not set ideally, it is easy to fall into the local optimal solution. According to the characteristics of the optimal solution, if a good initial value can be set, the optimal solution can be found quickly without falling into the local optimal solution. Therefore, it is necessary to read certain parameters from the camera manual during subsequent calibration, generally f,s x s, y , because these three parameters are relatively fixed and depend on the camera's craftsmanship.

[0051] S3, using the rigid transformation matrix solved in S2, sets the reference camera and unifies the cameras into the same coordinate system through calculation

[0052] In the present invention, the coordinates of the center of the marker point in the world coordinate system are set as P w (x,y,z,1) T , and the center coordinates of the same marker point in camera 0 are The coordinates in camera 1 are Both have been obtained through calibration, and the transformation of the relative pose of the two cameras is still a rigid transformation.

[0053] The rigid transformation between the two is:

[0054]

[0055] So we have:

[0056]

[0057] Rewrite the above formula as:

[0058]

[0059] Furthermore, the external parameters of the camera, namely R and t in the above formula, are known through the previous calibration, and the camera 0 is set as the reference coordinate, that is, You can find The above method can also be used to calculate the position of camera 3 relative to camera 0.

[0060] For camera 2 and camera 1, repeat step S2 to obtain the transformation relationship of the relative pose matrix

[0061]

[0062] Write the above formula as

[0063]

[0064] The relative pose relationship between camera 1 and camera 2 can be obtained. Then the relative pose relationship between camera 2 and camera 0 can be obtained.

[0065] Repeat the above steps to obtain the relative position relationship between camera 4 and camera 0.

[0066] At this point, all cameras are unified into the coordinate system of camera 0, completing the camera calibration for this example.

[0067] It should be understood that this example shows the calibration of five cameras, and the calibration of more cameras is also feasible using this method.

[0068] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A multi-view visual calibration method, characterized in that: include: Step 1: Build a multi-camera system using a calibration plate with known landmark parameters. The field of view of at least two cameras can simultaneously cover one calibration plate image. The field of view of a single camera in the multi-camera system can cover the entire calibration plate. In the multi-camera system, adjacent cameras observe at least one calibration image simultaneously, ensuring that the relative pose of adjacent cameras can be calculated from the calibration image. Step 2: Use image morphology correlation algorithms to find the pixel coordinates of the landmarks, transform the calibration plate pose, collect different calibration images to find the optimal solution, and solve the internal and external parameters of the camera, including focal length, distortion coefficient, scale factor, and the rigid transformation matrix of the relative pose, as follows: Step S21: extract the landmark points and calculate their pixel coordinates using the formula: Where (X c ,Y c ,Z c ) represents pixel coordinates, (X w ,Y w ,Z w ) represents the world coordinate, R is the 3x3 orthogonal rotation matrix, t is the three-dimensional translation vector, O T For the 0 vector matrix of corresponding size, calculate the values ​​of R and t; Step S22: Capture multiple images of the calibration plate and calculate the optimal solution d(θ) for the camera parameters using the following formula: The coordinates of the center of the dot obtained by the calibration plate are M j , extract the center coordinates of the camera calibration image, the first camera is recorded as m j,k,1 , the second camera is denoted as m j,k,2 , the Nth camera is recorded as m j,k,l , and θ is used to represent the parameters of the two cameras, including the internal parameters and relative positions of the two cameras, where π i Represents the projection of a certain mark point on the calibration plate to the image plane coordinate system, θ i,l is the subset of camera parameters in camera l that affect this projection; π r Indicates the correction of a point in the image to the image plane coordinate system, θ r,l represents the subset of parameters in camera l that affect this correction; v j,k,l Indicates whether this point is valid; n0 is the number of calibration images; n m is the number of landmark points; Step S23: Set any one of the cameras as a reference camera, i.e., the coordinate origin, and calculate the internal and external parameters of the camera; Step 3: Use the solved rigid transformation matrix to set the reference camera and unify the cameras into the same coordinate system through calculation. Specifically: Step S31: The world coordinate point P of the calibration plate with the same posture observed by the adjacent cameras in step S22 w (x,y,z,1) T , then the transformation relationship between two adjacent cameras is: in and The world coordinate system of the calibration plate with the same pose observed by different cameras is obtained by solving the relevant parameters to obtain the transformation relationship of the relative pose, where R is the rotation matrix and t is the translation increment; Step S32: Based on the relative pose relationships of different cameras calculated in S31, the pose relationships of all cameras in the system are further calculated.

2. A multi-view visual calibration system, characterized in that: For implementing the method according to claim 1, the system comprises: In a multi-camera system, cameras are numbered 0-N. A calibration plate with known landmark parameters is used, and the fields of view of at least two cameras can simultaneously cover one calibration plate image. The field of view of a single camera in the camera system can cover the entire calibration plate. In a multi-camera system, adjacent cameras simultaneously observe at least one calibration image, ensuring that the relative poses of adjacent cameras can be calculated from the calibration images. The calibration module uses image morphology-related algorithms to find the pixel coordinates of the landmark points, transform the calibration plate posture, collect different calibration images to find the optimal solution, and solve the internal and external parameters of the camera, including focal length, distortion coefficient, scale factor, and the rigid transformation matrix of the relative posture; using the solved rigid transformation matrix, set the reference camera and unify the cameras into the same coordinate system through calculation.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to claim 1 are implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 1 are implemented.

Images