A multi-camera calibration method without common field of view based on line structured light

Abstract

The present invention relates to a multi-camera calibration method without a common field of view based on line structured light, comprising: calibrating the internal parameters of each camera using planar calibration devices in at least three different directions in space; adjusting the laser plane so that the spatial calibration device receives clear laser lines within the effective imaging area of two adjacent cameras, wherein each camera receives at least two sets of imaging; extracting the energy center coordinates of the laser lines from the captured laser line images, and calculating a fitting plane within the respective camera coordinate system using at least two laser lines for each camera; repeating the above steps to obtain multiple sets of fitting plane parameters; and using the plane coplanarity constraint based on the obtained multiple sets of fitting plane parameters to solve the initial relative position relationship between the cameras, and improving the calibration accuracy through the LM optimization algorithm. The present invention can provide high-precision calibration results at a low cost and in more scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a multi-camera calibration method without a common field of view based on line structured light. Background Art

[0002] Calibration is the most basic and important part in the field of machine vision and computer vision. Through camera calibration, the internal parameters of the camera and the relative position relationship between cameras can be obtained. Calibration is the basis for converting two-dimensional image information into spatial physical information. The accuracy of calibration determines the effectiveness and accuracy of the visual measurement system.

[0003] The combined application of multiple cameras is increasingly playing a role in large-scale measurement, video surveillance, 3D reconstruction and other fields. In industrial applications, cameras are distributed in different positions in space according to needs. When the multi-camera combination does not have a common field of view, the traditional planar target calibration method cannot be applied.

[0004] Currently, multi-camera calibration without a common field of view is primarily accomplished using methods such as precision turntables, large-scale calibration devices, and theodolites. However, the accuracy of precision turntables and theodolites is constrained by cost: the higher the accuracy, the higher the cost. The economic cost required to obtain high-precision calibration results is prohibitive. Furthermore, the precision turntable method is not suitable for situations where the baseline distance between camera groups is large. Large-scale calibration devices require high material stiffness to prevent deformation due to their large size. This limits the available materials for the calibration device, and due to their size, the calibration cost is very high. Furthermore, the feasibility of calibration is limited by the application scenario, requiring a large, barrier-free calibration space, thus presenting certain limitations. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multi-camera calibration method without a common field of view based on line structured light, which can provide high-precision calibration results in more scenarios at a lower cost.

[0006] The technical solution adopted by the present invention to solve the technical problem is to provide a multi-camera calibration method without a common field of view based on line structured light, comprising the following steps:

[0007] (1) Using a planar calibration device in at least three different directions in space to calibrate the internal parameters of each camera;

[0008] (2) Adjust the laser plane so that the spatial calibration device receives clear laser lines within the effective imaging area of two adjacent cameras, wherein each camera receives at least two sets of imaging times;

[0009] (3) In the captured laser line image, the energy center coordinates of the laser line are extracted, and the fitting plane is calculated in the respective camera coordinate systems using no less than two laser lines for each camera;

[0010] (4) Repeat steps (2)-(3) to obtain at least three sets of fitting plane parameters;

[0011] (5) Based on the obtained multiple sets of fitting plane parameters, the plane coplanarity constraint is used to solve the initial relative position relationship between the cameras, and the LM optimization algorithm is used to improve the calibration accuracy.

[0012] The step (1) specifically includes:

[0013] For each camera in the multi-camera setup, place a calibration plate within the camera's field of view and capture the checkerboard pattern on the calibration plate while ensuring that it is fully and clearly imaged. The calibration plate must be placed in at least three different locations.

[0014] Extract the characteristic pattern on the calibration plate and establish an image coordinate system Ouv, where the origin O of the image coordinate system Ouv is the upper left corner pixel of the camera sensor; establish the camera coordinate system Ouv c X c Y c Z c , the camera coordinate system O c X c Y c Z c Origin O c is the optical center of the camera, Z in the camera coordinate system c The axis coincides with the optical axis of the camera and intersects with the image coordinate system Ouv; establish the world coordinate system O w X w Y w Z w The internal parameters of the camera are calibrated according to Zhang Zhengyou's calibration method, and the internal parameter equation of the camera is obtained: Among them, s represents the image physical coordinate system conversion coefficient, k c is the second-order radial distortion coefficient, r is the distance from the discrete point on the characteristic pattern to the intersection of the sensor and the optical axis of the camera, [uv1] T is the coordinate of the discrete point on the feature pattern in the image coordinate system Ouv, [X c Y c Z c ] T The discrete points on the feature pattern are in the camera coordinate system O c X c Y c Z c Lower coordinate, f x ,fy ,c x ,c y are the internal parameters of the camera.

[0015] The step (2) specifically includes:

[0016] Adjust the laser plane so that the spatial calibration plate receives clear laser lines within the effective imaging area of two adjacent cameras;

[0017] For the first camera of the two adjacent cameras, turn on the line laser and adjust the exposure of the first camera so that the first camera captures an image of the spatial calibration plate with a clear laser line pattern, and record the exposure as E L Turn off the line laser and adjust the exposure of the first camera so that the first camera can capture the spatial calibration pattern with clear contrast. Record the exposure as E H ;

[0018] Place the space calibration plate near the working distance of the first camera of the two adjacent cameras, turn on the line laser, and adjust the exposure of the first camera to E L , get image I L ; Turn off the line laser and adjust the exposure of the first camera to E H , get image I H ; Repeat this step to obtain multiple sets of images from the first camera;

[0019] When the positions of the line laser and the two adjacent cameras remain unchanged, the above method is used to take pictures with the second camera of the two adjacent cameras to obtain multiple groups of images of the second camera.

[0020] The step (3) specifically includes:

[0021] Select an image I from a set of images from the plurality of sets of images from the first camera H , extract and locate image I H The coordinates of the image coordinate system Ouv are converted to the image I according to the internal parameters of the camera. H Convert from image coordinate system Ouv to camera coordinate system O c X c Y c Z c Under the coordinates, determine the space calibration plate in the world coordinate system O w X w Y w Z w The coordinates under the space calibration plate are based on the characteristic position of the pattern and the image I H The one-to-one correspondence between the feature positions on the camera is obtained by c Xc Y c Z c To the world coordinate system O w X w Y w Z w The translation matrix T between i and the rotation matrix R i , where i represents the i-th spatial calibration plate;

[0022] For an image I in a set of selected images L , extract image I L The energy center ridge of the laser strip is used to filter the light strips that fall on the plane of the space calibration plate, and the image coordinates of the light strip point set are recorded. According to the internal parameters of the camera and the translation matrix T i and the rotation matrix R i , calculate the coordinates of the light strip point set in the camera coordinate system;

[0023] Apply the above steps to each of the multiple sets of images from the first camera to obtain a laser plane point set in the camera coordinate system, perform plane fitting on the laser plane point set, and obtain the plane equation of the laser plane in the first camera coordinate system: n1·p-d1=0, where p is the three-dimensional coordinate of any point on the laser plane in the first camera coordinate system, n1 is the unit vector of the normal direction of the laser plane in the first camera coordinate system, and d1 is the normal distance between the laser plane and the origin of the first camera coordinate system;

[0024] Repeat the above steps for the second camera to obtain the plane equation of the same laser plane in the second camera coordinate system: n2·p′-d2=0, where p′ is the three-dimensional coordinate of any point on the laser plane in the second camera coordinate system, n2 is the unit vector of the normal direction of the laser plane in the second camera coordinate system, and d2 is the normal distance between the laser plane and the origin of the second camera coordinate system.

[0025] The step (5) is specifically as follows:

[0026] According to multiple sets of fitting plane parameters, the plane parameter set in the coordinate systems of two adjacent cameras is obtained: Wherein, k represents the kth set of fitting plane parameters;

[0027] According to the plane coplanarity constraint, calculate the translation vector T from the second camera to the first camera c The initial solution of : Where N1 is a 3×N matrix consisting of N sets of plane unit normal vectors in the first camera coordinate system, and D1 and D2 are two N×1 matrices consisting of the normal distances from the origin of the first camera coordinate system and the origin of the second camera coordinate system to the plane.

[0028] According to the plane coplanarity constraint, calculate the rotation matrix R from the second camera to the first camera c The initial solution of : Among them, USV T is a matrix Obtained through singular value decomposition;

[0029] According to the product of the vector formed by the origin of the camera coordinate system and the point on the laser plane and the normal vector of the laser plane, which is equal to the normal distance from the camera coordinate system to the plane, the projection difference formula is established: Where P2 is the three-dimensional coordinate of the laser point in the coordinate system of the second camera;

[0030] The average projection error is obtained according to the projection difference formula and used as the optimization equation:

[0031] Translate the second camera to the first camera by the vector T c The initial solution and the rotation matrix R from the second camera to the first camera c The initial solution of is used as the initial value, and the optimization equation is used as the optimization condition. The LM optimization algorithm is used to minimize the average projection error and obtain the optimal solution of the translation vector and rotation matrix from the second camera to the first camera.

[0032] Beneficial effects

[0033] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention can ensure the accuracy and high precision of external parameter calibration between cameras at a relatively low cost in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a multi-camera calibration method without a common field of view based on line structured light according to an embodiment of the present invention;

[0035] Figure 2 3 is a spatial schematic diagram of a multi-camera calibration method without a common field of view based on line structured light according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0037] The embodiments of the present invention relate to a multi-camera calibration method without a common field of view based on line structured light, such as Figure 1 and Figure 2 As shown in the figure, it calibrates the camera's intrinsic parameters, adjusts the laser plane, captures and extracts laser lines, performs plane fitting in the coordinate systems of two adjacent cameras, and obtains the relative position relationship between the cameras through the constraint of the coplanar relationship between the planes. Finally, the calibration accuracy is optimized using the Levenberg-Marquarelt algorithm.

[0038] The main process of this embodiment can be implemented in C++ language, using OpenCV library and Ceres library as auxiliary, and specifically includes the following steps:

[0039] Step 1: Use a plane calibration device in at least three different directions in space to calibrate the internal parameters of each camera. The implementation process is as follows:

[0040] (11) For each camera in the multi-camera, place a calibration plate in the field of view of the camera and take photos while ensuring that the checkerboard pattern on the calibration plate can be imaged completely and clearly, wherein the calibration plate is placed in at least three different positions;

[0041] (12) Extract the characteristic pattern on the calibration plate and establish an image coordinate system Ouv, where the origin O of the image coordinate system Ouv is the upper left corner pixel of the sensor of the camera; establish the camera coordinate system Ouv c X c Y c Z c , the camera coordinate system O c X c Y c Z c Origin O c is the optical center of the camera, Z in the camera coordinate system c The axis coincides with the optical axis of the camera and intersects with the image coordinate system Ouv; establish the world coordinate system O w X w Y w Z w ;The internal parameters of the camera were calibrated according to Zhang Zhengyou calibration method.

[0042] Among them, the intrinsic parameter equation of the camera is: Among them, s represents the image physical coordinate system conversion coefficient, k c is the second-order radial distortion coefficient, r is the distance from the discrete point on the characteristic pattern to the intersection of the sensor and the optical axis of the camera, [uv1] T is the coordinate of the discrete point on the feature pattern in the image coordinate system Ouv, [X c Y c Z c ] TThe discrete points on the feature pattern are in the camera coordinate system O c X c Y c Z c Lower coordinate, f x ,f y ,c x ,c y are the internal parameters of the camera.

[0043] Step 2: Adjust the laser plane so that the spatial calibration device receives clear laser lines within the effective imaging area of two adjacent cameras. Each camera receives at least two sets of images. The implementation process is as follows:

[0044] (21) Adjusting the laser plane so that the spatial calibration plate receives clear laser lines within the effective imaging area of two adjacent cameras;

[0045] (22) For the first camera of the two adjacent cameras, turn on the line laser and adjust the exposure of the first camera so that the first camera captures an image of the space calibration plate with a clear laser line pattern, and record the exposure as E L ;

[0046] (23) For the first camera of the two adjacent cameras, turn off the line laser and adjust the exposure of the first camera so that the first camera can capture a spatial calibration pattern with clear contrast. The exposure is recorded as E H ;

[0047] (24) Place the space calibration plate near the working distance of the first camera of the two adjacent cameras, turn on the line laser, and adjust the exposure of the first camera to E L , get image I L ;

[0048] (25) Do not move the spatial calibration plate, turn off the line laser, and adjust the exposure of the first camera to E H , get image I H ;

[0049] (26) Repeat steps (24)-(25), with each camera capturing at least two sets of images, preferably more than 10 sets of images;

[0050] (27) When the positions of the online laser and the two adjacent cameras remain unchanged, the above steps are used to take pictures with the second camera of the two adjacent cameras to obtain multiple sets of images of the second camera.

[0051] Step 3: Extract the energy center coordinates of the laser lines from the captured laser line images. Use at least two laser lines from each camera to calculate the fitting plane in their respective camera coordinate systems. The detailed process is as follows:

[0052] (31) Select an image I from a group of images from the plurality of groups of images of the first camera. H , extract and locate image I H The coordinates of the image coordinate system Ouv are converted to the image I according to the internal parameters of the camera. H Convert from image coordinate system Ouv to camera coordinate system O c X c Y c Z c Under the coordinates, determine the space calibration plate in the world coordinate system O w X w Y w Z w The coordinates under the space calibration plate are based on the characteristic position of the pattern and the image I H The one-to-one correspondence between the feature positions on the camera is obtained by c X c Y c Z c To the world coordinate system O w X w Y w Z w The translation matrix T between i and the rotation matrix R i , where i represents the i-th spatial calibration plate;

[0053] (32) For image I in a selected set of images L , extract image I L The energy center ridge of the laser strip is used to filter the light strips that fall on the plane of the space calibration plate, and the image coordinates of the light strip point set are recorded. According to the internal parameters of the camera and the translation matrix T i and the rotation matrix R i , calculate the coordinates of the light strip point set in the camera coordinate system;

[0054] (33) Applying the above steps to each of the multiple sets of images of the first camera to obtain a laser plane point set in the camera coordinate system, performing plane fitting on the laser plane point set, and obtaining a plane equation of the laser plane in the first camera coordinate system: n1·p-d1=0, wherein p is the three-dimensional coordinate of any point on the laser plane in the first camera coordinate system, n1 is the unit vector of the normal direction of the laser plane in the first camera coordinate system, and d1 is the normal distance between the laser plane and the origin of the first camera coordinate system;

[0055] (34) For the second camera, repeat the above steps to obtain the plane equation of the same laser plane in the second camera coordinate system: n2·p′-d2=0, where p′ is the three-dimensional coordinate of any point on the laser plane in the second camera coordinate system, n2 is the unit vector of the normal direction of the laser plane in the second camera coordinate system, and d2 is the normal distance between the laser plane and the origin of the second camera coordinate system.

[0056] Step 4: Repeat steps 2-3 to obtain at least three sets of fitting plane parameters. More than five sets of fitting plane parameters are optimal.

[0057] In step 5, based on the obtained multiple sets of fitting plane parameters, the initial relative position relationship between the cameras is solved using the plane coplanarity constraint, and the calibration accuracy is improved through the LM optimization algorithm. The detailed implementation process is as follows:

[0058] (51) Based on multiple sets of fitting plane parameters, the plane parameter set in the coordinate systems of two adjacent cameras is obtained: Wherein, k represents the kth set of fitting plane parameters;

[0059] (52) According to the plane coplanarity constraint, calculate the translation vector T from the second camera to the first camera c The initial solution of : Where N1 is a 3×N matrix consisting of N sets of plane unit normal vectors in the first camera coordinate system, and D1 and D2 are two N×1 matrices consisting of the normal distances from the origin of the first camera coordinate system and the origin of the second camera coordinate system to the plane.

[0060] (53) According to the plane coplanarity constraint, calculate the rotation matrix R from the second camera to the first camera c The initial solution of : Among them, USV T is the matrix N1N1 T Obtained through singular value decomposition;

[0061] (54) Since the laser point is located on the laser plane, the product of the vector formed by the origin of the camera coordinate system and the point on the laser plane and the normal vector of the laser plane is equal to the normal distance from the camera coordinate system to the plane, and the projection difference formula is established: Where P2 is the three-dimensional coordinate of the laser point in the coordinate system of the second camera;

[0062] (55) According to the projection difference formula, the average projection error is obtained and used as the optimization equation:

[0063] (56) Translate the second camera to the first camera by the translation vector T c The initial solution and the rotation matrix R from the second camera to the first camerac The initial solution of is used as the initial value, and the optimization equation is used as the optimization condition. The LM optimization algorithm is used to minimize the average projection error and obtain the optimal solution of the translation vector and rotation matrix from the second camera to the first camera.

[0064] Step 6: Repeat steps 2 to 5 to calibrate the external parameters between any two adjacent cameras in the multi-camera system, and obtain the translation vector and rotation matrix between any two adjacent cameras.

[0065] It is not difficult to find that the present invention can ensure the accuracy and high precision of external parameter calibration between cameras at a relatively low cost in complex scenes.

Claims

1. A multi-camera calibration method without common field of view based on line structured light, characterized in that: The following steps are involved: (1) Using a planar calibration device in at least three different directions in space to calibrate the internal parameters of each camera; (2) Adjust the laser plane so that the spatial calibration device receives clear laser lines within the effective imaging area of two adjacent cameras, wherein each camera receives at least two sets of imaging times; (3) In the captured laser line image, the energy center coordinates of the laser line are extracted, and the fitting plane is calculated in the respective camera coordinate systems using no less than two laser lines for each camera; (4) Repeat steps (2)-(3) to obtain at least three sets of fitting plane parameters; (5) Based on the obtained multiple sets of fitting plane parameters, the initial relative position relationship between the cameras is solved using the plane coplanarity constraint, and the calibration accuracy is improved through the LM optimization algorithm. Specifically: According to multiple sets of fitting plane parameters, the plane parameter set in the coordinate systems of two adjacent cameras is obtained: Wherein, k represents the kth set of fitting plane parameters, n1 is the unit vector of the normal direction of the laser plane in the first camera coordinate system, d1 is the normal distance between the laser plane and the origin of the first camera coordinate system, n2 is the unit vector of the normal direction of the laser plane in the second camera coordinate system, and d2 is the normal distance between the laser plane and the origin of the second camera coordinate system; According to the plane coplanarity constraint, calculate the translation vector T from the second camera to the first camera c The initial solution of : Where N1 is a 3×N matrix consisting of N sets of plane unit normal vectors in the first camera coordinate system, and D1 and D2 are two N×1 matrices consisting of the normal distances from the origin of the first camera coordinate system and the origin of the second camera coordinate system to the plane. According to the plane coplanarity constraint, calculate the rotation matrix R from the second camera to the first camera c The initial solution of : Among them, USV T is a matrix Obtained through singular value decomposition; According to the product of the vector formed by the origin of the camera coordinate system and the point on the laser plane and the normal vector of the laser plane, which is equal to the normal distance from the camera coordinate system to the plane, the projection difference formula is established: E=(n1 T (R c P2+T c )-d1) 2 , where P2 is the three-dimensional coordinate of the laser point in the second camera coordinate system; The average projection error is obtained according to the projection difference formula and used as the optimization equation: Translate the second camera to the first camera by the vector T c The initial solution and the rotation matrix R from the second camera to the first camera c The initial solution of is used as the initial value, and the optimization equation is used as the optimization condition. The LM optimization algorithm is used to minimize the average projection error and obtain the optimal solution of the translation vector and rotation matrix from the second camera to the first camera.

2. The multi-camera calibration method without common field of view based on line structured light according to claim 1, characterized in that: Step (1) specifically includes: For each camera in the multi-camera setup, place a calibration plate within the camera's field of view and capture the checkerboard pattern on the calibration plate while ensuring that it is fully and clearly imaged. The calibration plate must be placed in at least three different locations. Extract the characteristic pattern on the calibration plate and establish an image coordinate system Ouv, where the origin O of the image coordinate system Ouv is the upper left corner pixel of the camera sensor; establish the camera coordinate system Ouv c X c Y c Z c , the camera coordinate system O c X c Y c Z c Origin O c is the optical center of the camera, Z in the camera coordinate system c The axis coincides with the optical axis of the camera and intersects with the image coordinate system Ouv; establish the world coordinate system O w X w Y w Z w The internal parameters of the camera are calibrated according to Zhang Zhengyou's calibration method, and the internal parameter equation of the camera is obtained: Among them, s represents the image physical coordinate system conversion coefficient, k c is the second-order radial distortion coefficient, r is the distance from the discrete point on the characteristic pattern to the intersection of the camera sensor and the optical axis, [uv 1] T is the coordinate of the discrete point on the feature pattern in the image coordinate system Ouv, [X c ,Y c ,Z c ] T The discrete points on the feature pattern are in the camera coordinate system O c X c Y c Z c Lower coordinate, f x 、f y 、c x and c y are the internal parameters of the camera.

3. The multi-camera calibration method without common field of view based on line structured light according to claim 2, characterized in that: Step (2) specifically includes: Adjust the laser plane so that the spatial calibration plate receives clear laser lines within the effective imaging area of two adjacent cameras; For the first camera of the two adjacent cameras, turn on the line laser and adjust the exposure of the first camera so that the first camera captures an image of the spatial calibration plate with a clear laser line pattern, and record the exposure as E L Turn off the line laser and adjust the exposure of the first camera so that the first camera can capture the spatial calibration pattern with clear contrast. Record the exposure as E H ; Place the space calibration plate near the working distance of the first camera of the two adjacent cameras, turn on the line laser, and adjust the exposure of the first camera to E L , get image I L ; Turn off the line laser and adjust the exposure of the first camera to E H , get image I H ; Repeat this step to obtain multiple sets of images from the first camera; When the positions of the line laser and the two adjacent cameras remain unchanged, the above method is used to take pictures with the second camera of the two adjacent cameras to obtain multiple groups of images of the second camera.

4. The multi-camera calibration method without common field of view based on line structured light according to claim 3, characterized in that: Step (3) specifically includes: Select an image I from a set of images from the plurality of sets of images from the first camera H , extract and locate image I H The coordinates of the image coordinate system Ouv are converted to the image I according to the internal parameters of the camera. H Convert from image coordinate system Ouv to camera coordinate system O c X c Y c Z c Under the coordinates, determine the space calibration plate in the world coordinate system O w X w Y w Z w The coordinates under the space calibration plate are based on the characteristic position of the pattern and the image I H The one-to-one correspondence between the feature positions on the camera is obtained by c X c Y c Z c To the world coordinate system O w X w Y w Z w The translation matrix T between i and the rotation matrix R i , where i represents the i-th spatial calibration plate; For an image I in a set of selected images L , extract image I L The energy center ridge of the laser strip is used to filter the light strips that fall on the plane of the space calibration plate, and the image coordinates of the light strip point set are recorded. According to the internal parameters of the camera and the translation matrix T i and the rotation matrix R i , calculate the coordinates of the light strip point set in the camera coordinate system; Apply the above steps to each of the multiple sets of images from the first camera to obtain a laser plane point set in the camera coordinate system, perform plane fitting on the laser plane point set, and obtain a plane equation of the laser plane in the first camera coordinate system: n1·p-d1=0, where p is the three-dimensional coordinate of any point on the laser plane in the first camera coordinate system; For the second camera, repeat the above steps to obtain the plane equation of the same laser plane in the second camera coordinate system: n2·p′-d2=0, where p′ is the three-dimensional coordinate of any point on the laser plane in the second camera coordinate system.

Images