Camera production line calibration method and system based on monocular three-dimensional reconstruction, verification method and system thereof, computer equipment and storage medium

Abstract

The invention belongs to the technical field of camera calibration and verification, and particularly relates to a camera production line calibration method and system based on monocular three-dimensional reconstruction, a verification method and system thereof, computer equipment and a storage medium. Adopting a Zhang Zhengyou calibration method to obtain initial calibration parameters of the camera; measuring the distance from the optical center of the camera to a certain feature point, obtaining the distances from the optical center to all the feature points according to the geometric constraint relationship, and reconstructing to obtain world coordinates of all the feature points; optimizing the initial calibration parameters of the camera by taking minimization of the feature point reconstruction error as a target to obtain the calibration parameters of the camera; and reconstructing the to-be-measured feature points in the camera coordinate system, calculating the distance between the reconstructed to-be-measured feature points, and comparing the distance with the actual distance to obtain a distance measurement error. The method is used for solving the problem that the re-projection error cannot accurately reflect the camera calibration precision, and the camera calibration experiment and the distance measurement verification experiment need to build experiment environments respectively, so that the calibration verification efficiency of the camera production line is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera calibration and verification, and specifically relates to a camera production line calibration method and system based on monocular three-dimensional reconstruction, and a verification method and system thereof, computer equipment and storage medium. Background Art

[0002] With the development of machine vision and artificial intelligence technology, image reconstruction and perception of the surrounding environment (3D vision) have become hot issues in academia and industry, and are widely used in video surveillance, visual navigation, industrial inspection, remote sensing and other fields. The goal of camera calibration is to obtain an accurate mathematical model that can accurately describe the conversion relationship between the camera's image coordinate system and the world coordinate system, so that in subsequent computer vision and machine vision applications, the content in the image can be accurately mapped into three-dimensional space, or accurate measurement and analysis can be performed.

[0003] At present, the camera calibration parameters are generally optimized by minimizing the reprojection error on the image plane. However, in applications such as machine vision, photogrammetry, and three-dimensional reconstruction, the spatial position of the object needs to be obtained. The reprojection error in pixels cannot directly reflect the calibration accuracy of the camera. At the same time, there may be overfitting problems and it cannot be applied to downstream reconstruction, ranging and other tasks. The existing technology relies on the feature points of the image taken by the binocular camera to match, and reconstructs the three-dimensional coordinates with the help of geometric relationships to minimize the three-dimensional reconstruction error and optimize the solution of internal and external parameters. This method is extremely dependent on the accuracy of binocular view matching and has poor robustness. At the same time, the camera parameters of the optimal reconstruction of a single camera need to be obtained in the camera calibration of the production line. Therefore, it is difficult to use the binocular camera for three-dimensional reconstruction in the camera production process. In order to more intuitively verify the camera calibration results, the existing technology also outputs the ranging error in millimeters through triangulation to reflect the accuracy of the calibration parameters in ranging. However, the parameter calibration process and the ranging verification experiment process need to be separately built in the experimental environment, and the test efficiency is low, which cannot meet the needs of camera production line calibration verification. Summary of the invention

[0004] The first purpose of the present invention is to provide a camera production line calibration method based on monocular three-dimensional reconstruction. On the existing production line camera calibration equipment, the camera calibration is completed by minimizing the reconstruction error, which solves the shortcomings of the existing camera calibration method and improves the camera calibration accuracy.

[0005] The present invention is achieved through the following technical solutions:

[0006] A camera production line calibration method based on monocular 3D reconstruction, the method comprising the following steps:

[0007] Step 1: Use Zhang Zhengyou calibration method to obtain the initial calibration parameters of the camera;

[0008] Step 2: Measure the distance from the camera optical center to a certain feature point, obtain the distance from the optical center to all feature points based on the geometric constraint relationship, and reconstruct the world coordinates of all feature points;

[0009] Step 3: Optimize the initial camera calibration parameters with the goal of minimizing the feature point reconstruction error to obtain the camera calibration parameters.

[0010] Furthermore, the step 1 specifically includes the following steps:

[0011] Step 1.1: Use the camera to be calibrated to take multiple images of the calibration plate and obtain the pixel coordinates of the feature points;

[0012] Step 1.2: Establish a mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system, calibrate the camera according to the mapping relationship, and obtain the initial calibration parameters of the camera.

[0013] Furthermore, the step 2 specifically includes the following steps:

[0014] Step 2.1: performing a dedistortion operation on the image based on the initial calibration parameters of the camera to obtain pixel coordinates of the dedistorted feature points;

[0015] Step 2.2: Obtaining a feature point direction vector based on the pixel coordinates of the feature point, and calculating the angle between the direction vectors of different feature points;

[0016] Step 2.3: Measure the distance from the optical center to a certain feature point, and use the angle of the direction vector of the feature point and the triangle constraint relationship to obtain the distance from all feature points to the optical center;

[0017] Step 2.4: Solve the depth information of each feature point and reconstruct the world coordinates of all feature points based on the depth information.

[0018] Furthermore, the step 2.1 is specifically as follows: assuming that the pixel coordinates of the feature point image coordinate system mdist detected on the distorted image are (udist, vdist), and the imaging coordinate system coordinates of the feature point are (x dist ,y dist ), on the undistorted image, the pixel coordinate m of the image coordinate system corresponding to the feature point is (u, v), and the coordinate of the imaging coordinate system is (x, y);

[0019] Dedistortion operation is performed through the dedistortion model to obtain the pixel coordinate m of the dedistorted image coordinate system as (u, v);

[0020] The dedistortion model includes a radial distortion model and a tangential distortion model.

[0021] Furthermore, the step 2.2 is specifically to perform a reverse projection transformation from the image coordinate system to the camera coordinate system according to the calibrated initial intrinsic parameters K of the camera and the pixel coordinates m of the dedistorted image coordinate system to obtain the normalized camera coordinate system coordinates P of the feature points. c ' ik ,

[0022] The normalized camera coordinate system coordinates of the feature point are the optical center O c Direction vector to the feature point

[0023] According to the direction vector of the feature point Unit vector with respect to the optical axis Use the vector angle cosine formula to get the cosine value of the angle between the feature point direction vector and the optical axis cos(α ik );

[0024] Calculate the direction vectors of the kth feature point and the lth feature point of the i-th image from the optical center respectively According to the cosine formula of the vector angle, the cosine of the angle between the feature points is obtained (θ kl ).

[0025] Furthermore, the step 2.3 is specifically to use a high-precision distance measuring instrument to measure the optical center O c To a certain feature point M ik The distance d ik ;

[0026] According to the triangle constraint relationship, the distance from the optical center to other feature points M is calculated. il The distance d il ; In triangle ΔO c M ik M il In, O c M ik The length d ik , the distance L between feature points kl And the cosine of the angle between the feature point vectors (θ kl ) is known, the cosine theorem can be used to solve for d il .

[0027] Furthermore, the step 2.4 is specifically as follows: according to the obtained optical center O c To a certain feature point M ik The distance is d ik , the cosine of the angle between the feature point and the optical axis; the feature point M can be obtained using the projection formula ik Depth information Z cik , according to the inverse projection formula, the world coordinates M of the feature point can be reconstructed i 'k .

[0028] Furthermore, the step 3 is specifically as follows: based on the reconstructed feature point M i ' j With the real feature point M ij The internal parameters and distortion parameters are optimized iteratively with the goal of minimizing the distance between them.

[0029] Compared with the prior art, the camera production line calibration method based on monocular 3D reconstruction created by the present invention has the following advantages:

[0030] The present invention realizes monocular 3D reconstruction on the existing camera production line calibration equipment, and further optimizes the calibration parameters with the goal of minimizing the reconstruction error, thereby improving the camera calibration accuracy.

[0031] The second purpose of the present invention is to provide a camera production line calibration system based on monocular 3D reconstruction, so as to implement a camera production line calibration method based on monocular 3D reconstruction.

[0032] The present invention is achieved through the following technical solutions:

[0033] A camera production line calibration system based on monocular 3D reconstruction, the system uses the camera production line calibration method based on monocular 3D reconstruction as described above, the system comprises:

[0034] The camera calibration module uses Zhang Zhengyou's calibration method to obtain the initial calibration parameters of the camera;

[0035] The feature point reconstruction module performs a dedistortion operation on the image based on the initial calibration parameters of the camera to obtain the pixel coordinates of the dedistorted feature points;

[0036] Obtain the direction vector of the feature point based on the pixel coordinates of the feature point, and calculate the angle between the direction vectors of different feature points;

[0037] According to the distance from the optical center to a certain feature point, the distance from all feature points to the optical center is obtained by using the angle of the feature point direction vector and the triangle constraint relationship;

[0038] Solve the depth information of each feature point and reconstruct the world coordinates of the feature point based on the depth information;

[0039] The parameter optimization module optimizes the initial calibration parameters of the camera with the goal of minimizing the feature point reconstruction error to obtain the camera calibration parameters;

[0040] Compared with the prior art, the camera production line calibration system based on monocular 3D reconstruction created by the present invention has the following advantages:

[0041] Aiming at the problem that monocular 3D reconstruction cannot be achieved on existing camera production line calibration equipment, the present invention designs a camera production line calibration system based on monocular 3D reconstruction, thereby minimizing the reconstruction error, further optimizing the calibration parameters, and improving the camera calibration accuracy.

[0042] The third purpose of the present invention is to provide a verification method for camera production line calibration based on monocular 3D reconstruction, so as to solve the problem that the parameter calibration process and the ranging verification experiment process need to be separately built in the experimental environment, and the need for verification after calibration on the camera production line cannot be met.

[0043] The present invention is achieved through the following technical solutions:

[0044] A verification method for camera production line calibration based on monocular 3D reconstruction, the verification method includes using the camera production line calibration method based on monocular 3D reconstruction as described above, the verification method specifically comprises:

[0045] After obtaining the camera calibration parameters, the feature points to be measured are reconstructed in the camera coordinate system, the distance between the reconstructed feature points to be measured is calculated, and the distance measurement error is obtained by comparing with the actual distance.

[0046] Further, a set of calibration plate images are re-photographed, and the distortion-free pixel coordinates of the feature points to be measured are obtained using the de-distortion method and the obtained distortion coefficients, and the camera coordinate system coordinates of the feature points to be measured are reconstructed using the obtained camera calibration parameters and the distortion coefficients;

[0047] The reconstructed camera coordinate system coordinates are used to obtain the distance between the feature points to be measured, and compared with the actual distance of the feature points to be measured to obtain the relative error of the distance measurement.

[0048] Furthermore, the camera coordinate system coordinates of the feature point to be measured are M c ' ik , M c ' il ,

[0049]

[0050] Use the reconstructed camera coordinate system coordinates to get the measured distance L k ' l ,

[0051] L k ' l =|M c ' ik -M c ' il | (16)

[0052] L k ' l The actual distance L from the feature pointkl Compare and get the relative error ε of ranging,

[0053]

[0054] After the ranging is completed, the calibration parameters, reprojection error, and relative ranging error are output.

[0055] Compared with the prior art, the verification method of camera production line calibration based on monocular 3D reconstruction created by the present invention has the following advantages:

[0056] The present invention completes the distance measurement verification experiment based on the camera production line calibration environment, simplifies the test process, and improves the calibration verification efficiency of the camera production line.

[0057] The fourth object of the present invention is to provide a verification system for camera production line calibration based on monocular 3D reconstruction, so as to implement a verification method for camera production line calibration based on monocular 3D reconstruction.

[0058] The present invention is achieved through the following technical solutions:

[0059] A verification system for camera production line calibration based on monocular 3D reconstruction, the system uses the verification method for camera production line calibration based on monocular 3D reconstruction as described above, the system comprises:

[0060] The parameter verification module reconstructs the feature points to be measured in the camera coordinate system, calculates the distance between the reconstructed feature points to be measured, and compares it with the actual distance to obtain the distance measurement error;

[0061] The working process of the parameter verification module is specifically as follows: the distortion-free pixel coordinates of the feature points to be measured are obtained using the distortion coefficients obtained by the dedistortion method and the parameter optimization module, and the camera coordinate system coordinates of the feature points to be measured are reconstructed using the camera calibration parameters and the distortion coefficients obtained by the parameter optimization module.

[0062] The reconstructed camera coordinate system coordinates are used to obtain the distance between the feature points to be measured, and compared with the actual distance of the feature points to be measured to obtain the relative error of the distance measurement.

[0063] Compared with the prior art, the camera production line calibration verification system based on monocular 3D reconstruction created by the present invention has the following advantages:

[0064] The present invention completes the distance measurement verification experiment for the existing camera production line calibration environment, simplifies the test process, improves the camera production line calibration verification efficiency and verifies the effectiveness of the calibration method of the present invention.

[0065] The fifth object of the present invention is to provide a computer device for running the step algorithm flow of the camera production line calibration method based on monocular three-dimensional reconstruction or the camera production line calibration verification method based on monocular three-dimensional reconstruction.

[0066] The present invention is achieved through the following technical solutions:

[0067] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the camera production line calibration method based on monocular three-dimensional reconstruction or the camera production line calibration verification method based on monocular three-dimensional reconstruction is implemented.

[0068] Compared with the prior art, the computer device described in the present invention has the following advantages:

[0069] The computer device of the present invention runs a calibration method based on monocular three-dimensional reconstruction for an existing camera production line, and runs a verification method based on the calibration method for an existing camera production line.

[0070] The sixth object of the present invention is to provide a storage medium for storing the step algorithm flow of the camera production line calibration method based on monocular three-dimensional reconstruction or the camera production line calibration verification method based on monocular three-dimensional reconstruction.

[0071] The present invention is achieved through the following technical solutions:

[0072] A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the camera production line calibration method based on monocular three-dimensional reconstruction as described above or implements the camera production line calibration verification method based on monocular three-dimensional reconstruction as described above.

[0073] Compared with the prior art, the storage medium created by the present invention has the following advantages:

[0074] The computer-readable storage medium of the present invention stores a calibration method based on monocular three-dimensional reconstruction for an existing camera production line, and stores a verification method based on the calibration method for an existing camera production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a method flow chart of the camera production line calibration method based on monocular three-dimensional reconstruction of the present invention.

[0076] Figure 2 It is a method flow chart of the verification method of the camera production line calibration based on monocular three-dimensional reconstruction of the present invention.

[0077] Figure 3 It is a schematic diagram of the characteristic point direction vector diagram of the present invention.

[0078] Figure 4 It is a schematic diagram of the angle between the characteristic point direction vector and the optical axis of the present invention.

[0079] Figure 5 It is a schematic diagram of the angle between the direction vectors of the characteristic points of the present invention.

[0080] Figure 6 It is a schematic diagram of the triangle constraint relationship of the present invention.

[0081] Figure 7 It is a schematic diagram of the verification method of camera calibration of the present invention.

[0082] Figure 8 It is a system schematic diagram of a camera production line calibration system based on monocular 3D reconstruction of the present invention.

[0083] Fig. 9 It is a system schematic diagram of a verification system for camera production line calibration based on monocular 3D reconstruction of the present invention.

[0084] Fig.10 It is a schematic diagram of a computer device of the present invention. DETAILED DESCRIPTION

[0085] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0086] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0087] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0088] The following is attached to this application specification Figure 1-10 , the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0089] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0090] Embodiment 1

[0091] The embodiment of the present invention provides a camera production line calibration method based on monocular 3D reconstruction, the method comprising the following steps:

[0092] Step 1: Use Zhang Zhengyou calibration method to obtain the initial calibration parameters of the camera;

[0093] Furthermore, the step 1 specifically includes the following steps:

[0094] Step 1.1: Use the camera to be calibrated to take multiple images of the calibration plate and obtain the pixel coordinates of the feature points;

[0095] The calibration board pattern includes but is not limited to one or a combination of checkerboard, circular spots, grid, BMW pattern and ChArUco; use a checkerboard calibration board with known number of rows, columns and feature point distances. The calibration board should be as flat as possible and the calibration pattern should be as clear as possible. Take multiple images of the calibration board in different poses so that the feature points cover the entire field of view of the camera as much as possible. This example uses a checkerboard calibration board with 14 rows and 20 columns of feature points and a feature point spacing of 60 mm. 17 calibration images are taken with the calibration board located in the center, top, bottom, left, right, top left, bottom left, top right, and bottom right of the image;

[0096] The Harris corner detection algorithm is used to extract the sub-pixel coordinates of the feature points. In this embodiment, the sub-pixel corner refinement window size is set to (11, 11), the number of iterations is 30, and the iteration accuracy is 0.001.

[0097] Furthermore, the step 1.1 also includes if there are problems such as uneven lighting, camera defocus, and excessive image noise during the calibration plate image capture process;

[0098] The reprojection error of each image is calculated based on the initial calibration, and abnormal images with too large reprojection errors are eliminated; the calibration plate image of the eliminated image pose is captured, and the camera is recalibrated.

[0099] Step 1.2: Establish the mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system, calibrate the camera according to the mapping relationship, and obtain the initial calibration parameters of the camera;

[0100] By obtaining the homography relationship between the world coordinates and pixel coordinates of the feature points on multiple images, the projection equation is established to solve the parameters of the pinhole camera model and construct the world coordinate system (O w -X w Y w Z w ), camera coordinate system (O c -X c Y c Z c ), imaging coordinate system (O-xy) and image coordinate system (O pix -uv) between the feature point. According to the pinhole model, the relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system coordinate m can be expressed as:

[0101]

[0102] In the formula, Z c is the depth information, f x , f y are the focal lengths of the camera in the x and y directions of the imaging coordinate system, u0 and v0 are the principal point offsets of the camera in the x and y directions of the imaging coordinate system, and K is the intrinsic parameter matrix. R and T are the rotation matrix and translation vector in the extrinsic parameters, respectively;

[0103] According to the camera distortion model,

[0104]

[0105] In the formula, (x dist ,y dist ) is the coordinate of the distorted feature point in the imaging coordinate system, (x, y) is the coordinate of the ideal undistorted feature point in the imaging coordinate system, where k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients. The initial solution of the internal and external parameters is solved through the homography relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system pixel coordinate m. Then, the solved internal and external parameters and distortion coefficients are globally optimized with the goal of minimizing the reprojection error;

[0106] The expression for minimizing the reprojection error is:

[0107]

[0108] In the formula, m ij is the pixel coordinate of the jth (j=1,···,m) feature point on the ith (i=1,···,n) captured image, m i ' j are the pixel coordinates of the reprojected feature points calculated according to the camera geometry model.

[0109] This embodiment uses the Levenberg-Marquardt optimization algorithm for iterative solution, the number of iterations is set to 500 times, and the iteration convergence accuracy is set to 1e-8.

[0110] Step 2: Measure the distance from the camera optical center to a certain feature point, obtain the distance from the optical center to all feature points based on the geometric constraint relationship, and reconstruct the world coordinates of all feature points;

[0111] Step 2.1: Dedistort the image based on the initial calibration parameters of the camera in step 1 to obtain the pixel coordinates of the feature points after dedistortion;

[0112] Specifically, step 2.1 is as follows: assuming that the pixel coordinates mdist of the feature point image coordinate system detected on the distorted image are (u dist ,v dist ), the imaging coordinate system coordinate of the feature point is (x dist ,y dist ), on the undistorted image, the pixel coordinate m of the image coordinate system corresponding to the feature point is (u, v), and the coordinate of the imaging coordinate system is (x, y);

[0113] Dedistortion operation is performed through the dedistortion model to obtain the pixel coordinate m of the dedistorted image coordinate system as (u, v);

[0114] According to the conversion relationship between the image coordinate system and the imaging coordinate system, we can get:

[0115]

[0116] In the formula, f x , f y are the focal lengths of the camera in the x and y directions of the imaging coordinate system, respectively; u0 and v0 are the principal point offsets of the camera in the x and y directions of the imaging coordinate system.

[0117] The obtained (x dist ,y dist ) as the initial value (x0,y0),

[0118]

[0119] The dedistorted imaging coordinate system (x, y) can be obtained by repeatedly iterating the distortion model.

[0120]

[0121] When the number of iterations or the iteration accuracy requirement is reached, the iteration is exited. n ,y n As the imaging coordinate system coordinates (x, y) on the undistorted image.

[0122] According to the conversion relationship between the image coordinate system and the imaging coordinate system,

[0123]

[0124] Through the above process, the pixel coordinate m of the dedistorted image coordinate system can be obtained as (u, v);

[0125] The dedistortion model includes a radial distortion model and a tangential distortion model.

[0126] Step 2.2: Obtaining a feature point direction vector based on the pixel coordinates of the feature point, and calculating the angle between the direction vectors of different feature points;

[0127] Furthermore, the step 2.2 is specifically to perform a reverse projection transformation from the image coordinate system to the camera coordinate system according to the calibrated initial intrinsic parameters K of the camera and the pixel coordinates m of the dedistorted image coordinate system to obtain the normalized camera coordinate system coordinates P of the feature points. c ' ik ,

[0128]

[0129] The normalized camera coordinate system coordinates of the feature point are the optical center O c Direction vector to the feature point

[0130] According to the direction vector of the feature point Unit vector with respect to the optical axis Use the vector angle cosine formula to get the cosine value of the angle between the feature point direction vector and the optical axis cos(α ik ),

[0131]

[0132] Calculate the direction vectors of the kth feature point and the lth feature point of the i-th image from the optical center respectively According to the cosine formula of the vector angle, the cosine of the angle between the feature points is obtained (θ kl ),

[0133]

[0134] Step 2.3: Measure the distance from the optical center to a certain feature point, and use the angle of the feature point direction vector and the triangle constraint relationship to obtain the distance from all feature points to the optical center;

[0135] Furthermore, the step 2.3 is specifically to use a high-precision distance measuring instrument to measure the optical center O c To a certain feature point M ik The distance d ik; In this example, the distance from the optical center to the first characteristic point measured by a laser rangefinder is 2185 mm;

[0136] According to the triangle constraint relationship, the distance from the optical center to other feature points M is calculated. il The distance d il ; In triangle ΔO c M ik M il In, O c M ik The length d ik , the distance L between feature points kl And the cosine of the angle between the feature point vectors (θ kl ) is known, the cosine theorem can be used to solve for d il ,

[0137]

[0138] Step 2.4: Calculate the depth information of each feature point by the distance between each feature point and the optical center, and reconstruct the world coordinates of all feature points based on the depth information;

[0139] Furthermore, the step 2.4 is specifically as follows: according to the obtained optical center O c To a certain feature point M ik The distance is d ik , the cosine of the angle between the feature point and the optical axis; the feature point M can be obtained using the projection formula ik Depth information Z cik ,

[0140] Z cik =d ik cos(α ik ) (12)

[0141] According to the inverse projection formula, the world coordinates M of the feature points can be reconstructed i ' k ,

[0142]

[0143] Step 3: Optimize the initial camera calibration parameters with the goal of minimizing the feature point reconstruction error to obtain the camera calibration parameters;

[0144] Furthermore, the step 3 is specifically to reconstruct the feature points M i ' j With the real feature point M ij The internal parameters and distortion parameters are optimized iteratively with the goal of minimizing the distance between them.

[0145]

[0146] This embodiment adopts the Levenberg-Marquardt optimization algorithm to optimize the initial calibration parameters of the camera with the goal of minimizing the feature point reconstruction error, and the number of iterations is set to 200 times.

[0147] The present embodiment also includes a verification method for camera production line calibration based on monocular three-dimensional reconstruction, and the verification method also includes: after obtaining the camera calibration parameters, reconstructing the feature points to be measured in the camera coordinate system, calculating the distance between the reconstructed feature points to be measured, and comparing with the actual distance to obtain the ranging error.

[0148] The verification method is specifically as follows: in order to avoid data overfitting, a set of calibration plate images are re-shot, the distortion-free pixel coordinates of the feature points to be measured are obtained using the dedistortion method of step 2.1 and the distortion coefficients obtained in step 3, and the camera coordinate system coordinates of the feature points to be measured are reconstructed using the camera calibration parameters and distortion coefficients obtained in step 3.

[0149] The reconstructed camera coordinate system coordinates are used to obtain the distance between the feature points to be measured, and compared with the actual distance of the feature points to be measured to obtain the relative error of the distance measurement.

[0150] Furthermore, the camera coordinate system coordinates of the point to be measured in the verification method are M c ' ik , M c ' il ,

[0151]

[0152] Use the reconstructed camera coordinate system coordinates to get the measured distance L k ' l ,

[0153] L k ' l =|M c ' ik -M c ' il | (16)

[0154] L' kl The actual distance L from the feature point kl The relative error ε of the distance measurement is obtained by comparison. The actual distance between the upper left corner feature point and the lower right corner feature point of the chessboard calibration plate measured in this embodiment is 1381.304 mm.

[0155]

[0156] After the ranging is completed, the calibration parameters, reprojection error, and relative ranging error are output.

[0157] Specifically,

[0158] Comparison table of experimental results of camera production line calibration method based on monocular 3D reconstruction

[0159]

[0160]

[0161] Embodiment 2

[0162] An embodiment of the present invention provides a camera production line calibration system based on monocular 3D reconstruction. The system uses the camera production line calibration method based on monocular 3D reconstruction as described in Embodiment 1. The system includes:

[0163] The initial calibration module uses Zhang Zhengyou's calibration method to obtain the initial calibration parameters of the camera;

[0164] The specific working process of the camera initial calibration module is to use the camera to be calibrated to shoot multiple calibration plate images to obtain the pixel coordinates of the feature points; the calibration plate patterns include but are not limited to chessboard, circular spots, field grid, BMW pattern and one or a combination of ChArUco; use a chessboard calibration plate with known number of rows, columns and feature point distances. The calibration plate should be as flat as possible and the calibration pattern should be as clear as possible. Take multiple calibration plate images in different postures so that the feature points cover the entire field of view of the camera as much as possible. This example uses a chessboard calibration plate with 14 rows and 20 columns of feature points and a feature point spacing of 60 mm. 17 calibration images are taken with the calibration plate located in the center, top, bottom, left, right, top left, bottom left, top right, and bottom right of the image;

[0165] The Harris corner detection algorithm is used to extract the sub-pixel coordinates of the feature points. In this embodiment, the sub-pixel corner refinement window size is set to (11, 11), the number of iterations is 30, and the iteration accuracy is 0.001.

[0166] Furthermore, it also includes the problem of uneven lighting, camera defocus, and excessive image noise during the calibration plate image capture process;

[0167] The reprojection error of each image is calculated based on the initial calibration, and abnormal images with too large reprojection errors are eliminated; the calibration plate image of the eliminated image pose is captured, and the camera is recalibrated.

[0168] Furthermore, a mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system is established, and the camera is calibrated according to the mapping relationship to obtain the initial calibration parameters of the camera;

[0169] By obtaining the homography relationship between the world coordinates and pixel coordinates of the feature points on multiple images, the projection equation is established to solve the parameters of the pinhole camera model and construct the world coordinate system (O w -Xw Y w Z w ), camera coordinate system (O c -X c Y c Z c ), imaging coordinate system (O-xy) and image coordinate system (O pix -uv) between the feature point. According to the pinhole model, the relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system coordinate m can be expressed as:

[0170]

[0171] In the formula, Z c is the proportionality coefficient, f x ,f y are the focal lengths of the camera in the x and y directions of the imaging coordinate system, u0 and v0 are the principal point offsets of the camera in the x and y directions of the imaging coordinate system, and K is the intrinsic parameter matrix. R and T are the rotation matrix and translation vector in the extrinsic parameters, respectively;

[0172] Furthermore, the camera distortion model is specifically:

[0173]

[0174] In the formula, (x dist ,y dist ) is the coordinate of the distorted feature point in the imaging coordinate system, (x, y) is the coordinate of the ideal undistorted feature point in the imaging coordinate system, where k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients. The initial solution of the internal and external parameters is solved through the homography relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system pixel coordinate m. Then, the solved internal and external parameters and distortion coefficients are globally optimized with the goal of minimizing the reprojection error;

[0175] The expression for minimizing the reprojection error is:

[0176]

[0177] In the formula, m ij is the pixel coordinate of the jth (j=1,···,m) feature point on the ith (i=1,···,n) captured image, m i ' j are the pixel coordinates of the reprojected feature points calculated according to the camera geometry model.

[0178] This embodiment uses the Levenberg-Marquardt optimization algorithm for iterative solution, the number of iterations is set to 500 times, and the iteration convergence accuracy is set to 1e-8.

[0179] The feature point reconstruction module performs a dedistortion operation on the image based on the initial calibration of the camera parameters to obtain the pixel coordinates of the dedistorted feature points;

[0180] Furthermore, let the pixel coordinates m of the feature point image coordinate system detected on the distorted image be dist is dist ,v dist ), the imaging coordinate system coordinate of the feature point is (x dist ,y dist ), on the undistorted image, the pixel coordinate m of the image coordinate system corresponding to the feature point is (u, v), and the coordinate of the imaging coordinate system is (x, y);

[0181] According to the conversion relationship between the image coordinate system and the imaging coordinate system, we can get:

[0182]

[0183] In the formula, f x ,f y are the focal lengths of the camera in the x and y directions of the imaging coordinate system, respectively; u0 and v0 are the principal point offsets of the camera in the x and y directions of the imaging coordinate system.

[0184] The obtained (x dist ,y dist ) as the initial value (x0,y0),

[0185]

[0186] The dedistorted imaging coordinate system (x, y) can be obtained by repeatedly iterating the distortion model.

[0187]

[0188] When the number of iterations or the iteration accuracy requirement is reached, the iteration is exited. n ,y n As the imaging coordinate system coordinates (x, y) on the undistorted image.

[0189] According to the conversion relationship between the image coordinate system and the imaging coordinate system,

[0190]

[0191] Through the above process, the pixel coordinate m of the dedistorted image coordinate system can be obtained as (u, v);

[0192] The dedistortion model includes a radial distortion model and a tangential distortion model.

[0193] Obtain the direction vector of the feature point based on the pixel coordinates of the feature point, and calculate the angle between the direction vectors of different feature points;

[0194] Furthermore, according to the calibrated camera initial intrinsic parameters K and the pixel coordinates m of the dedistorted image coordinate system, the inverse projection transformation from the image coordinate system to the camera coordinate system is performed to obtain the normalized camera coordinate system coordinates P of the feature points. c ' ik ,

[0195]

[0196] The normalized camera coordinate system coordinates of the feature point are the optical center O c Direction vector to the feature point

[0197] According to the direction vector of the feature point Unit vector with respect to the optical axis Use the vector angle cosine formula to get the cosine value of the angle between the feature point direction vector and the optical axis cos(α ik ),

[0198]

[0199] Calculate the direction vectors of the kth feature point and the lth feature point of the i-th image from the optical center respectively According to the cosine formula of the vector angle, the cosine of the angle between the feature points is obtained (θ kl ),

[0200]

[0201] According to the distance from the optical center to a certain feature point, the distance from all feature points to the optical center is obtained by using the angle between the direction vectors of different feature points and the triangle constraint relationship;

[0202] Furthermore, a high-precision distance measuring instrument is used to measure the optical center O c To a certain feature point M ik The distance d ik ; In this embodiment, the distance from the optical center to the first characteristic point measured by a laser rangefinder is 2185 mm;

[0203] According to the triangle constraint relationship, the distance from the optical center to other feature points M is calculated. il The distance d il ; In triangle ΔO c M ik M il In, O c M ik The length d ik , the distance L between feature pointskl And the cosine of the angle between the feature point vectors (θ kl ) is known, the cosine theorem can be used to solve for d il ,

[0204]

[0205] The depth information of each feature point is solved by the distance between each feature point and the optical center, and the world coordinates of all feature points are reconstructed based on the depth information;

[0206] Furthermore, according to the obtained optical center O c To a certain feature point M ik The distance is d ik , the cosine of the angle between the feature point and the optical axis; the feature point M can be obtained using the projection formula ik Depth information Z cik ,

[0207] Z cik =d ik cos(α ik ) (12)

[0208] According to the inverse projection formula, the world coordinates M of the feature points can be reconstructed i ' k ,

[0209]

[0210] The parameter optimization module optimizes the initial camera calibration parameters with the goal of minimizing the feature point reconstruction error to obtain the camera calibration parameters.

[0211] Further, according to the reconstructed feature point M i ' j With the real feature point M ij The internal parameters and distortion parameters are optimized iteratively with the goal of minimizing the distance between them.

[0212]

[0213] This embodiment adopts the Levenberg-Marquardt optimization algorithm to optimize the initial calibration parameters of the camera with the goal of minimizing the feature point reconstruction error, and the number of iterations is set to 200 times.

[0214] This embodiment also includes a verification system for camera production line calibration based on monocular 3D reconstruction, the verification system comprising:

[0215] The parameter verification module reconstructs the feature points to be measured in the camera coordinate system, calculates the distance between the reconstructed feature points to be measured, and compares it with the actual distance to obtain the distance measurement error;

[0216] The working process of the parameter verification module is specifically as follows: the distortion-free pixel coordinates of the feature points to be measured are obtained using the distortion coefficients obtained by the dedistortion method and the parameter optimization module, and the camera coordinate system coordinates of the feature points to be measured are reconstructed using the camera calibration parameters and the distortion coefficients obtained by the parameter optimization module.

[0217] The reconstructed camera coordinate system coordinates are used to obtain the distance between the feature points to be measured, and compared with the actual distance of the feature points to be measured to obtain the relative error of the distance measurement.

[0218] Furthermore, the camera coordinate system coordinates of the feature point to be measured in step 7 are M c ' ik , M c ' il ,

[0219]

[0220] Use the reconstructed camera coordinate system coordinates to get the measured distance L k ' l ,

[0221] L k ' l =|M c ' ik -M c ' il | (16)

[0222] L k ' l The actual distance L from the feature point kl The relative error ε of the distance measurement is obtained by comparison. The actual distance between the upper left corner feature point and the lower right corner feature point of the chessboard calibration plate measured in this embodiment is 1381.304 mm.

[0223]

[0224] After the ranging is completed, the calibration parameters, reprojection error, and relative ranging error are output.

[0225] Embodiment 3

[0226] An embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory is used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected via a bus. Specifically, the processor implements any step in the first embodiment above by running the computer program stored in the memory.

[0227] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0228] The memory may include a read-only memory, a flash memory, and a random access memory, and provides instructions and data to the processor. A part or all of the memory may also include a nonvolatile random access memory.

[0229] As can be seen from the above, the electronic device provided by the embodiment of the present invention can implement the camera production line calibration method and verification method based on monocular 3D reconstruction as described in Example 1 by running a computer program, so as to solve the problem that the parameter calibration process and the range verification experiment process need to be built separately in the experimental environment. It is not limited to a specific camera geometric model and is widely applicable to the calibration and verification of camera production lines with a variety of field of view ranges.

[0230] It should be understood that if the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0231] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0232] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0233] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.

[0234] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0235] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0236] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A camera production line calibration method based on monocular 3D reconstruction, characterized in that: The method comprises the following steps, Step 1: Use Zhang Zhengyou calibration method to obtain the initial calibration parameters of the camera; Step 2: Measure the distance from the camera optical center to a certain feature point, obtain the distance from the optical center to all feature points based on the geometric constraint relationship, and reconstruct the world coordinates of all feature points; Step 3: Optimize the initial camera calibration parameters with the goal of minimizing the feature point reconstruction error to obtain the camera calibration parameters.

2. The camera production line calibration method based on monocular 3D reconstruction according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1.1: Use the camera to be calibrated to take multiple images of the calibration plate and obtain the pixel coordinates of the feature points; Step 1.2: Establish a mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system, calibrate the camera according to the mapping relationship, and obtain the initial calibration parameters of the camera.

3. The camera production line calibration method based on monocular 3D reconstruction according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: performing a dedistortion operation on the image based on the initial calibration parameters of the camera to obtain pixel coordinates of the dedistorted feature points; Step 2.2: Obtaining a feature point direction vector based on the pixel coordinates of the feature point, and calculating the angle between the direction vectors of different feature points; Step 2.3: Measure the distance from the optical center to a certain feature point, and use the angle of the direction vector of the feature point and the triangle constraint relationship to obtain the distance from all feature points to the optical center; Step 2.4: Solve the depth information of each feature point and reconstruct the world coordinates of all feature points based on the depth information.

4. The camera production line calibration method based on monocular 3D reconstruction according to claim 3, characterized in that: Specifically, step 2.1 is as follows: assuming that the pixel coordinates m of the feature point image coordinate system detected on the distorted image are dist is dist ,v dist ), the imaging coordinate system coordinate of the feature point is (x dist ,y dist ), on the undistorted image, the pixel coordinate m of the image coordinate system corresponding to the feature point is (u, v), and the coordinate of the imaging coordinate system is (x, y); Dedistortion operation is performed through the dedistortion model to obtain the pixel coordinate m of the dedistorted image coordinate system as (u, v); The dedistortion model includes a radial distortion model and a tangential distortion model.

5. The camera production line calibration method based on monocular 3D reconstruction according to claim 3, characterized in that: Specifically, step 2.2 is to perform a reverse projection transformation from the image coordinate system to the camera coordinate system based on the calibrated initial camera intrinsic parameters K and the pixel coordinates m of the dedistorted image coordinate system to obtain the normalized camera coordinate system coordinates P of the feature points. c ' ik ; The normalized camera coordinate system coordinates of the feature point are the optical center O c Direction vector to the feature point According to the direction vector of the feature point Unit vector with respect to the optical axis Use the vector angle cosine formula to get the cosine value of the angle between the feature point direction vector and the optical axis cos(α ik ); Calculate the direction vectors of the kth feature point and the lth feature point of the i-th image from the optical center respectively According to the cosine formula of the vector angle, the cosine of the angle between the feature points is obtained (θ kl ).

6. The camera production line calibration method based on monocular 3D reconstruction according to claim 3, characterized in that: The step 2.3 specifically includes measuring the optical center O using a high-precision distance measuring instrument. c To a certain feature point M ik The distance d ik ; According to the triangle constraint relationship, the distance from the optical center to other feature points M is calculated. il The distance d il ; In triangle ΔO c M ik M il In, O c M ik The length d ik , the distance L between feature points kl And the cosine of the angle between the feature point vectors (θ kl ) is known, the cosine theorem can be used to solve for d il .

7. The camera production line calibration method based on monocular 3D reconstruction according to claim 3, characterized in that: The step 2.4 is specifically as follows: according to the obtained optical center O c To a certain feature point M ik The distance is d ik , the cosine of the angle between the feature point and the optical axis; the feature point M can be obtained using the projection formula ik Depth information Z cik , according to the inverse projection formula, the world coordinates M of the feature point can be reconstructed i ' k .

8. The camera production line calibration method based on monocular 3D reconstruction according to claim 1, characterized in that: The step 3 is specifically as follows: ij ′ and the real feature point M ij The internal parameters and distortion parameters are optimized iteratively with the goal of minimizing the distance between them.

9. A verification method for camera production line calibration based on monocular 3D reconstruction, characterized in that: The verification method includes using a camera production line calibration method based on monocular 3D reconstruction as described in any one of claims 1 to 8, and the verification method is specifically as follows: After obtaining the camera calibration parameters, the feature points to be measured are reconstructed in the camera coordinate system, the distance between the reconstructed feature points to be measured is calculated, and the distance measurement error is obtained by comparing with the actual distance.

10. The verification method for camera production line calibration based on monocular 3D reconstruction according to claim 9, characterized in that: Re-shoot a set of calibration plate images, use the de-distortion method and the obtained distortion coefficient to obtain the distortion-free pixel coordinates of the feature points to be measured, and use the obtained camera calibration parameters and distortion coefficient to reconstruct the camera coordinate system coordinates of the feature points to be measured; The reconstructed camera coordinate system coordinates are used to obtain the distance between the feature points to be measured, and compared with the actual distance of the feature points to be measured to obtain the relative error of the distance measurement.

11. The verification method for camera production line calibration based on monocular 3D reconstruction according to claim 9, characterized in that: The camera coordinate system coordinates of the feature point to be measured are M c ' ik , M c ' il , Use the reconstructed camera coordinate system coordinates to get the measured distance L k ' l , L k ′ l =|M c ′ ik -M c ′ il | (16) L k ' l The actual distance L from the feature point kl Compare and get the relative error ε of ranging, After the ranging is completed, the calibration parameters, reprojection error, and relative ranging error are output.

12. A camera production line calibration system based on monocular 3D reconstruction, characterized in that: The system uses the camera production line calibration method based on monocular three-dimensional reconstruction as described in any one of claims 1 to 8, and the system includes: The camera calibration module uses Zhang Zhengyou's calibration method to obtain the initial calibration parameters of the camera; The feature point reconstruction module performs a dedistortion operation on the image based on the initial calibration parameters of the camera to obtain the pixel coordinates of the dedistorted feature points; Obtain the direction vector of the feature point based on the pixel coordinates of the feature point, and calculate the angle between the direction vectors of different feature points; According to the distance from the optical center to a certain feature point, the distance from all feature points to the optical center is obtained by using the angle of the feature point direction vector and the triangle constraint relationship; Solve the depth information of each feature point and reconstruct the world coordinates of the feature point based on the depth information; The parameter optimization module optimizes the initial camera calibration parameters with the goal of minimizing the feature point reconstruction error to obtain the camera calibration parameters.

13. A verification system for camera production line calibration based on monocular 3D reconstruction, characterized in that: The system uses the verification method of camera production line calibration based on monocular three-dimensional reconstruction as claimed in claim 9, and the system includes: The parameter verification module reconstructs the feature points to be measured in the camera coordinate system, calculates the distance between the reconstructed feature points to be measured, and compares it with the actual distance to obtain the distance measurement error; The working process of the parameter verification module is specifically as follows: obtaining the distortion-free pixel coordinates of the feature point to be measured using the distortion coefficients obtained by the dedistortion method and the parameter optimization module, and reconstructing the camera coordinate system coordinates of the feature point to be measured using the camera calibration parameters and the distortion coefficients obtained by the parameter optimization module; The reconstructed camera coordinate system coordinates are used to obtain the distance between the feature points to be measured, and compared with the actual distance of the feature points to be measured to obtain the relative error of the distance measurement.

14. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 8 or the method according to claim 9 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 or the method according to claim 9 is implemented.

Images