Camera production line calibration method and system based on monocular three-dimensional reconstruction, verification method and system thereof, computer device and storage medium

Abstract

The application belongs to the technical field of camera calibration and verification, and particularly relates to a camera production line calibration method and system based on monocular three-dimensional reconstruction, a verification method and system thereof, a computer device and a storage medium. Zhang Zhengyou calibration method is adopted to obtain initial camera calibration parameters; the distance from the camera optical center to a certain feature point is measured, the distance from the optical center to all feature points is obtained according to the geometric constraint relationship, and the world coordinates of all feature points are reconstructed; the initial camera calibration parameters are optimized with the minimization of the feature point reconstruction error as the target to obtain the camera calibration parameters; the to-be-measured feature points are reconstructed in the camera coordinate system, the distance between the reconstructed to-be-measured feature points is calculated, and the ranging error is obtained by comparing with the actual distance. The application is used to solve the problem that the reprojection error cannot accurately reflect the camera calibration accuracy, and the camera calibration and ranging verification experiments need to be built respectively, thereby improving the camera production line calibration verification efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of camera calibration and verification, and particularly relates to a camera production line calibration method based on monocular three-dimensional reconstruction, a system thereof, a verification method and system thereof, a computer device and a storage medium. BACKGROUND

[0002] With the development of machine vision and artificial intelligence technology, image reconstruction and perception of the surrounding environment (3D vision) have become a hot issue in the academic and industrial fields and are widely used in video monitoring, visual navigation, industrial detection, remote sensing and other fields. The goal of camera calibration is to obtain an accurate mathematical model that can accurately describe the conversion relationship between the image coordinate system of the camera and the world coordinate system, so that the content in the image can be accurately mapped to the three-dimensional space or accurately measured and analyzed in subsequent computer vision and machine vision applications.

[0003] Currently, the camera calibration parameters are generally optimized by minimizing the re-projection error on the image plane. However, in the applications of machine vision, photogrammetry, three-dimensional reconstruction, etc., the spatial position of the object needs to be obtained, and the re-projection error in pixels cannot intuitively reflect the calibration accuracy of the camera, and there may be overfitting problems that cannot be applied to downstream reconstruction, ranging and other tasks. The existing technology relies on matching the corrected feature points of the images captured by the binocular camera and reconstructing the three-dimensional coordinates with the help of geometric relationships to optimize and solve the internal and external parameters by minimizing the three-dimensional reconstruction error. This method is extremely dependent on the accuracy of binocular view matching and has poor robustness. At the same time, in the production line camera calibration, the optimal reconstruction camera parameters of a single camera need to be obtained, so it is difficult to apply the binocular camera for three-dimensional reconstruction in the camera production process. In order to more intuitively verify the camera calibration results, the existing technology also outputs the ranging error in millimeters to reflect the accuracy of the calibration parameters in ranging, but the parameter calibration process and the ranging verification experiment process need to be built separately, which is low in testing efficiency and cannot meet the needs of camera production line calibration verification. SUMMARY

[0004] The first purpose of the present application is to provide a camera production line calibration method based on monocular three-dimensional reconstruction, which minimizes the reconstruction error to complete the calibration of the camera on the existing production line camera calibration equipment, solves the shortcomings of the existing camera calibration method, and improves the calibration accuracy of the camera.

[0005] The present application is achieved by the following technical solutions:

[0006] A camera production line calibration method based on monocular three-dimensional reconstruction, the method comprising the following steps,

[0007] Step 1: Zhang Zhengyou calibration method is used to obtain the initial calibration parameters of the camera;

[0008] Step 2: measuring the distance from the camera optical center to a certain feature point, obtaining the distance from the optical center to all feature points according to the geometric constraint relationship, and reconstructing the world coordinates of all feature points;

[0009] Step 3: optimizing the initial camera calibration parameters to minimize the feature point reconstruction error, and obtaining the camera calibration parameters.

[0010] Further, the step 1 specifically comprises the following steps,

[0011] Step 1.1: using the camera to be calibrated to shoot multiple calibration board images, and obtaining the pixel coordinates of the feature points;

[0012] Step 1.2: establishing the mapping relationship between the image coordinate system pixel coordinates and the feature point world coordinate system coordinates, calibrating the camera according to the mapping relationship, and obtaining the initial calibration parameters of the camera.

[0013] Further, the step 2 specifically comprises the following steps:

[0014] Step 2.1: performing a de-distortion operation on the image based on the initial camera calibration parameters to obtain the de-distorted feature point pixel coordinates;

[0015] Step 2.2: obtaining the feature point direction vector based on the feature point pixel coordinates, and calculating the included angle of different feature point direction vectors;

[0016] Step 2.3: measuring the distance from the optical center to a certain feature point, and obtaining the distance from the optical center to all feature points by using the included angle of the feature point direction vector and the triangle constraint relationship;

[0017] Step 2.4: solving the depth information of each feature point, and reconstructing the world coordinates of all feature points according to the depth information.

[0018] Further, the step 2.1 specifically is that the image coordinate system pixel coordinates mdist of the feature point detected on the distorted image is (udist, vdist), the imaging coordinate system coordinates of the feature point are (x dist ,y dist ), and the image coordinate system pixel coordinates m of the feature point on the undistorted image are (u, v) and the imaging coordinate system coordinates are (x, y);

[0019] The de-distorted image coordinate system pixel coordinates m obtained by the de-distortion model are (u, v);

[0020] The de-distortion model includes a radial distortion model and a tangential distortion model.

[0021] Further, step 2.2 specifically involves performing an inverse projection transformation from the image coordinate system to the camera coordinate system based on the calibrated initial intrinsic parameters K of the camera and the pixel coordinates m of the distortion-free image coordinate system to obtain the normalized camera coordinates P of the feature points. c ′ ik ,

[0022] The normalized camera coordinates of the feature points are the optical center O. c Direction vector to feature point

[0023] Based on the direction vector of the feature points unit vector relative to the optical axis The cosine value of the angle between the feature point direction vector and the optical axis, cos(α), is obtained using the formula for the cosine of the vector angle. ik );

[0024] Calculate the direction vectors originating from the optical center for the k-th and l-th feature points in the i-th image, respectively. The cosine of the angle between feature points, cos(θ), can be obtained using the formula for the cosine of the angle between vectors. kl ).

[0025] Furthermore, step 2.3 specifically involves using a high-precision ranging instrument to measure the optical center O. c To a certain feature point M ik distance d ik ;

[0026] Based on the triangular constraint relationship, the distance M from the optical center to other feature points is calculated. il distance d il In triangle ΔO c M ik M il In the middle, O c M ik Length d ik The distance L between feature points kl and the cosine of the angle between the feature point vectors, cos(θ) kl Given that d can be solved using the Law of Cosines. il .

[0027] Further, step 2.4 specifically involves, based on the obtained optical center O... c To a certain feature point M ik The distance is d ik The cosine of the angle between the obtained feature point and the optical axis; the feature point M can be obtained using the projection formula. ik Deep Information Z cik The world coordinates M of the feature point can be reconstructed using the inverse projection formula. i ′k .

[0028] Further, the step 3 is specifically, according to the reconstructed feature point M i ′ j The distance between the real feature point M ij The internal parameter and the distortion parameter are iteratively optimized with the minimum distance as the target.

[0029] Compared with the prior art, the camera production line calibration method based on monocular three-dimensional reconstruction has the following advantages:

[0030] The application realizes monocular three-dimensional reconstruction on the existing camera production line calibration device, and further optimizes the calibration parameters with the minimum reconstruction error as the target, thereby improving the camera calibration precision.

[0031] A second object of the application is to provide a camera production line calibration system based on monocular three-dimensional reconstruction, which realizes the camera production line calibration method based on monocular three-dimensional reconstruction.

[0032] The application realizes the above objects by the following technical solutions:

[0033] A camera production line calibration system based on monocular three-dimensional reconstruction, which uses the camera production line calibration method based on monocular three-dimensional reconstruction, and comprises,

[0034] A camera calibration module, which adopts Zhang Zhengyou's calibration method to obtain initial calibration parameters of the camera;

[0035] A feature point reconstruction module, which performs a de-distortion operation on the image based on the initial calibration parameters of the camera to obtain de-distorted feature point pixel coordinates;

[0036] A feature point direction vector is obtained based on the feature point pixel coordinates, and the included angle of different feature point direction vectors is calculated;

[0037] According to the measured distance from the optical center to a certain feature point, the included angle of the feature point direction vector and the triangular constraint relationship are used to obtain the distance from the optical center to all feature points;

[0038] The depth information of each feature point is solved, and the world coordinates of the feature points are reconstructed according to the depth information;

[0039] A parameter optimization module, which optimizes the initial calibration parameters of the camera with the minimum feature point reconstruction error as the target to obtain the camera calibration parameters;

[0040] Compared with the prior art, the camera production line calibration system based on monocular three-dimensional reconstruction has the following advantages:

[0041] The present application is directed to the problem that monocular three-dimensional reconstruction cannot be realized on the existing camera production line calibration device, and a camera production line calibration system based on monocular three-dimensional reconstruction is designed to minimize reconstruction error, further optimize calibration parameters, and improve camera calibration accuracy.

[0042] A third object of the present application is to provide a camera production line calibration verification method based on monocular three-dimensional reconstruction to solve the problem that the parameter calibration process and the ranging verification experiment process need to be built separately, and cannot meet the demand of verification after calibration on the camera production line.

[0043] The present application is achieved by the following technical solutions:

[0044] A camera production line calibration verification method based on monocular three-dimensional reconstruction, the verification method comprises using a camera production line calibration method based on monocular three-dimensional reconstruction as described above, and the verification method specifically comprises,

[0045] After obtaining the camera calibration parameters, the feature points to be measured are reconstructed in the camera coordinate system, the distance between the reconstructed feature points to be measured is calculated, and the ranging error is obtained by comparing with the actual distance.

[0046] Further, a group of calibration board images is re-shot, the distortion-free pixel coordinates of the feature points to be measured are obtained by using the distortion removal method and the obtained distortion coefficients, and the camera coordinate system coordinates of the feature points to be measured are reconstructed by using the obtained camera calibration parameters and distortion coefficients;

[0047] The distance between the feature points to be measured is obtained by using the reconstructed camera coordinate system coordinates, and the ranging relative error is obtained by comparing with the actual distance of the feature points to be measured.

[0048] Further, the camera coordinate system coordinates of the feature points to be measured are M c ′ ik , M c ′ il ,

[0049]

[0050] The distance to be measured L k ′ l ,

[0051] L k ′ l = |M c ′ ik -M c ′ il | (16)

[0052] L k ′ l is compared with the actual distance Lkl The ranging relative error epsilon is obtained by comparison,

[0053]

[0054] The calibration parameters, the re-projection error and the ranging relative error are outputted after the ranging.

[0055] Compared with the prior art, the verification method for camera production line calibration based on monocular three-dimensional reconstruction has the following advantages:

[0056] The ranging verification experiment is completed based on the camera production line calibration environment, the test process is simplified, and the camera production line calibration verification efficiency is improved.

[0057] A fourth object of the present application is to provide a verification system for camera production line calibration based on monocular three-dimensional reconstruction, which implements the verification method for camera production line calibration based on monocular three-dimensional reconstruction.

[0058] The present application is implemented by the following technical solutions:

[0059] A verification system for camera production line calibration based on monocular three-dimensional reconstruction, which uses the verification method for camera production line calibration based on monocular three-dimensional reconstruction as described above, and comprises,

[0060] The parameter verification module reconstructs the to-be-measured feature points in the camera coordinate system, calculates the distances between the reconstructed to-be-measured feature points, and compares the distances with the actual distances to obtain ranging errors;

[0061] The working process of the parameter verification module is specifically that the distortion-free pixel coordinates of the to-be-measured feature points are obtained by using the distortion coefficients obtained by the distortion removal method and the parameter optimization module, and the camera coordinate system coordinates of the to-be-measured feature points are reconstructed by using the camera calibration parameters and the distortion coefficients obtained by the parameter optimization module.

[0062] The distances between the to-be-measured feature points are obtained by using the reconstructed camera coordinate system coordinates, and the ranging relative error is obtained by comparing the distances with the actual distances of the to-be-measured feature points.

[0063] Compared with the prior art, the verification system for camera production line calibration based on monocular three-dimensional reconstruction has the following advantages:

[0064] The ranging verification experiment is completed based on the camera production line calibration environment, the test process is simplified, and the camera production line calibration verification efficiency is improved.

[0065] A fifth object of the present application is to provide a computer device for running the step algorithm process of the camera production line calibration method based on monocular three-dimensional reconstruction or the verification method for camera production line calibration based on monocular three-dimensional reconstruction.

[0066] The application is realized by the following technical scheme:

[0067] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the camera production line calibration method based on monocular three-dimensional reconstruction or the verification method of the camera production line calibration based on monocular three-dimensional reconstruction as described above when executing the computer program.

[0068] Compared with the prior art, the computer device of the application has the following advantages:

[0069] The computer device of the application runs the calibration method based on monocular three-dimensional reconstruction for an existing camera production line, and runs the verification method based on the calibration method for the existing camera production line.

[0070] The sixth object of the application is to provide a storage medium for storing the step algorithm flow of the camera production line calibration method based on monocular three-dimensional reconstruction or the verification method of the camera production line calibration based on monocular three-dimensional reconstruction.

[0071] The application is realized by the following technical scheme:

[0072] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the camera production line calibration method based on monocular three-dimensional reconstruction or the verification method of the camera production line calibration based on monocular three-dimensional reconstruction as described above.

[0073] Compared with the prior art, the storage medium of the application has the following advantages:

[0074] The computer readable storage medium of the application stores the calibration method based on monocular three-dimensional reconstruction for an existing camera production line, and stores the verification method based on the calibration method for the existing camera production line. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is a method flowchart of the camera production line calibration method based on monocular three-dimensional reconstruction of the application.

[0076] Figure 2 is a method flowchart of the verification method of the camera production line calibration based on monocular three-dimensional reconstruction of the application.

[0077] Figure 3 is a feature point direction vector diagram of the application.

[0078] Figure 4 is a feature point direction vector and optical axis angle diagram of the application.

[0079] Figure 5 is an angle diagram between feature point direction vectors of the present application.

[0080] Figure 6 is a triangle constraint relationship diagram of the present application.

[0081] Figure 7 is a camera calibration verification method diagram of the present application.

[0082] Figure 8 is a system diagram of a camera production line calibration system based on monocular three-dimensional reconstruction of the present application.

[0083] Figure 9 is a system diagram of a camera production line calibration verification system based on monocular three-dimensional reconstruction of the present application.

[0084] Figure 10 is a computer device diagram of the present application. DETAILED DESCRIPTION

[0085] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0086] It should be understood that the term "comprises" when used in this specification and the appended claims, specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0087] It should also be understood that the terms used in the specification and the appended claims are intended to describe particular embodiments and do not intend to limit the present application. As used in the specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0088] The embodiments of the present application will be described below with reference to the accompanying drawings. Figures 1-10 The technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0089] Many specific details are set forth in the following description in order to provide a thorough understanding of the present application. However, the present application can be practiced according to other embodiments that can not be described in detail herein, and the present application is not limited to the embodiments described herein. Rather, the present application is limited only by the claims.

[0090] Embodiment one

[0091] The embodiment of the present application provides a camera production line calibration method based on monocular three-dimensional reconstruction, and the method comprises the following steps,

[0092] Step 1: Obtain the initial calibration parameters of the camera by using Zhang Zhengyou calibration method;

[0093] Further, the step 1 specifically comprises the following steps,

[0094] Step 1.1: Obtain the pixel coordinates of the feature points by using the to-be-calibrated camera to shoot multiple calibration board images;

[0095] The calibration board pattern comprises but is not limited to one or a combination of several of the following: a checkerboard, a circle spot, a Chinese checkboard, a BMW pattern and ChArUco. A checkerboard calibration board with known row number, column number and feature point distance is used. The calibration board is as flat as possible, and the calibration pattern is as clear as possible. Multiple calibration board images in different poses are shot, so that the feature points can cover the entire field of view of the camera as much as possible. In this example, a checkerboard calibration board with 14 rows and 20 columns of feature points and a feature point spacing of 60 mm is used, and 17 calibration images in the poses of image center, top, bottom, left, right, upper left, lower left, upper right and lower right are shot;

[0096] The Harris corner point detection algorithm is used to extract the sub-pixel coordinates of the feature points. In this embodiment, the sub-pixel corner point refinement window size is set to (11, 11), the iteration number is 30 times, and the iteration accuracy is 0.001.

[0097] Further, the step 1.1 further comprises the problems of uneven illumination, camera defocus and excessive image noise in the calibration board image shooting process.

[0098] According to the initial calibration, the re-projection error of each image is calculated, and the abnormal images with excessive re-projection error are removed. The calibration board image in the removed image pose is shot, and the camera calibration is performed again.

[0099] Step 1.2: Establish the mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system, calibrate the camera according to the mapping relationship, and obtain the initial calibration parameters of the camera;

[0100] The projection equation is established by obtaining the homographic relationship between the world coordinates and the pixel coordinates of the feature points on the photographed multiple images to solve the parameters of the pinhole camera model, and the conversion relationship between the world coordinate system (O w -X w Y w Z w ), the camera coordinate system (O c -X c Y c Z c ), the imaging coordinate system (O-xy) and the image coordinate system (O pix -uv) is constructed. According to the pinhole model, the relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system coordinate m can be expressed as:

[0101]

[0102] In the formula, Z c is the depth information, f x , f y are the focal lengths of the camera in the x and y directions of the imaging coordinate system respectively, u0 and v0 are the principal point offsets of the camera in the x and y directions of the imaging coordinate system respectively, K is the intrinsic matrix. R and T are the rotation matrix and the translation vector in the extrinsic parameters respectively;

[0103] According to the camera distortion model,

[0104]

[0105] In the formula, (x dist ,y dist ) is the coordinate of the imaging coordinate system distortion feature point, (x, y) is the coordinate of the imaging coordinate system ideal non-distortion feature point, wherein k1, k2 and k3 are the radial distortion coefficients, and p1 and p2 are the tangential distortion coefficients. The initial solution of the intrinsic and extrinsic parameters is solved by the homographic relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system pixel coordinate m. The intrinsic and extrinsic parameters and the distortion coefficients solved are globally optimized by taking the minimization of the re-projection error as the target;

[0106] The expression of the minimization of the re-projection error is:

[0107]

[0108] In the formula, m ij is the pixel coordinate of the jth (j=1,..., m) feature point on the ith (i=1,..., n) photographed image, and m i ′ j is the pixel coordinate of the re-projected feature point calculated according to the camera geometric model.

[0109] The Levenberg-Marquardt optimization algorithm is used for iterative solution in this embodiment, the number of iterations is set to 500 times, and the iteration convergence accuracy is set to 1e-8.

[0110] Step 2: Measure the distance from the camera optical center to a certain feature point, obtain the distance from the optical center to all feature points according to the geometric constraint relationship, and reconstruct to obtain the world coordinates of all feature points;

[0111] Step 2.1: Perform de-distortion operation on the image based on the initial calibration parameters of the camera in step 1 to obtain the de-distorted feature point pixel coordinates;

[0112] The step 2.1 is specifically that the feature point image coordinate system pixel coordinates mdist detected on the distorted image is (u dist ,v dist ), the imaging coordinate system coordinates of the feature point are (x dist ,y dist ), and the image coordinate system pixel coordinates m of the feature point on the undistorted image are (u,v), and the imaging coordinate system coordinates are (x,y);

[0113] The de-distortion operation is performed through the de-distortion model to obtain the de-distorted image coordinate system pixel coordinates m as (u,v);

[0114] According to the conversion relationship between the image coordinate system and the imaging coordinate system, the following is obtained:

[0115]

[0116] In the formula, f x , f y are the focal lengths of the camera in the imaging coordinate system x and y directions respectively, and u0 and v0 are the principal point offsets of the camera in the imaging coordinate system x and y directions.

[0117] The obtained (x dist ,y dist ) is taken as the initial value (x0, y0),

[0118]

[0119] The de-distorted imaging coordinate system coordinates (x, y) can be obtained through repeated iteration of the distortion model,

[0120]

[0121] When the number of iterations or the iteration accuracy requirement is reached, the iteration is exited. The x n , y n obtained by iteration are taken as the imaging coordinate system coordinates (x, y) on the undistorted image.

[0122] This is obtained based on the transformation relationship between the image coordinate system and the imaging coordinate system.

[0123]

[0124] The above process can be used to obtain the pixel coordinates m (u, v) of the image coordinate system after distortion correction.

[0125] The distortion correction model includes a radial distortion model and a tangential distortion model.

[0126] Step 2.2: Obtain the feature point direction vector based on the feature point pixel coordinates, and calculate the angle between different feature point direction vectors;

[0127] Furthermore, step 2.2 specifically involves performing an inverse projection transformation from the image coordinate system to the camera coordinate system based on the calibrated initial intrinsic parameters K of the camera and the pixel coordinates m in the distortion-free image coordinate system to obtain the normalized camera coordinates P of the feature points. c ′ ik ,

[0128]

[0129] The normalized camera coordinates of the feature points are the optical center O. c Direction vector to feature point

[0130] Based on the direction vector of the feature points unit vector relative to the optical axis The cosine value of the angle between the feature point direction vector and the optical axis, cos(α), is obtained using the formula for the cosine of the vector angle. ik ),

[0131]

[0132] Calculate the direction vectors originating from the optical center for the k-th and l-th feature points in the i-th image, respectively. The cosine of the angle between feature points, cos(θ), can be obtained using the formula for the cosine of the angle between vectors. kl ),

[0133]

[0134] Step 2.3: Measure the distance from the optical center to a certain feature point, and use the angle between the direction vectors of the feature point and the triangular constraint relationship to obtain the distance from all feature points to the optical center;

[0135] Furthermore, step 2.3 specifically involves using a high-precision ranging instrument to measure the optical center O. c To a certain feature point M ik distance d ikIn this example, the distance from the optical center to the first feature point is measured by a laser range finder to be 2185mm;

[0136] The distance d il from the optical center to other feature points M il is calculated according to the triangle constraint relationship c In the triangle ΔO ik M il M c , the length d ik , the distance L ik between feature points, and the cosine cos(θ kl ) of the angle between feature point vectors are known, and d kl can be solved using the cosine law il ,

[0137]

[0138] Step 2.4: Solve the depth information of each feature point by the distance from each feature point to the optical center, and reconstruct the world coordinates of all feature points according to the depth information;

[0139] Further, step 2.4 is specifically, according to the distance d c from the optical center O ik to a feature point M ik , the cosine of the angle between the feature point and the optical axis is obtained; using the projection formula, the depth information Z ik of the feature point M cik can be obtained ,

[0140] Z cik = d ik cos(α ik ) (12)

[0141] According to the inverse projection formula, the world coordinates M i ′ k of the feature point can be reconstructed ,

[0142]

[0143] Step 3: Optimize the initial camera calibration parameters to minimize the feature point reconstruction error, and obtain the camera calibration parameters;

[0144] Further, step 3 is specifically, according to the minimum distance between the reconstructed feature point M i ′ j and the real feature point M ij , the internal parameters and distortion parameters are iteratively optimized.

[0145]

[0146] The embodiment adopts Levenberg-Marquardt optimization algorithm to optimize the initial camera calibration parameters with the target of minimizing the feature point reconstruction error, and the iteration number is set to 200 times.

[0147] The embodiment also includes a camera production line calibration verification method based on monocular three-dimensional reconstruction, and the verification method further includes: after obtaining the camera calibration parameters, reconstructing the to-be-measured feature points in the camera coordinate system, calculating the distance between the reconstructed to-be-measured feature points, and comparing the actual distance to obtain the ranging error.

[0148] The verification method specifically is that, in order to avoid the data overfitting phenomenon, a group of calibration board images are re-shot, the to-be-measured feature point undistorted pixel coordinates are obtained by using the distortion removal method of step 2.1 and the distortion coefficients obtained in step 3, and the camera coordinate system coordinates of the to-be-measured feature points are reconstructed by using the camera calibration parameters and the distortion coefficients obtained in step 3.

[0149] The distance between the to-be-measured feature points is obtained by using the reconstructed camera coordinate system coordinates, and the ranging relative error is obtained by comparing the actual distance of the to-be-measured feature points.

[0150] Further, the camera coordinate system coordinates of the to-be-measured points of the verification method are M c ′ ik , M c ′ il ,

[0151]

[0152] The to-be-measured distance L k ′ l is obtained by using the reconstructed camera coordinate system coordinates,

[0153] L k ′ l = |M c ′ ik -M c ′ il | (16)

[0154] The ranging relative error ε is obtained by comparing L' kl with the actual distance L kl of the feature points, and the actual distance of the upper left corner feature point and the lower right corner feature point of the checkerboard calibration board measured in the embodiment is 1381.304 mm,

[0155]

[0156] After the ranging is completed, the calibration parameters, the re-projection error, and the ranging relative error are output.

[0157] Specifically,

[0158] Table of experimental results of camera production line calibration method based on monocular three-dimensional reconstruction

[0159]

[0160]

[0161] Embodiment two

[0162] The embodiment of the application provides a camera production line calibration system based on monocular three-dimensional reconstruction, which uses the camera production line calibration method based on monocular three-dimensional reconstruction as described in embodiment one, and the system comprises,

[0163] An initial calibration module adopts Zhang Zhengyou calibration method to obtain initial calibration parameters of the camera.

[0164] The specific working process of the camera initial calibration module is that a plurality of calibration board images are shot by the camera to be calibrated to obtain pixel coordinates of feature points; the calibration board pattern includes but is not limited to one or a combination of several of a checkerboard, a circular spot, a Chinese checkboard, a BMW pattern and ChArUco; a checkerboard calibration board with known row number, column number and feature point distance is used. The calibration board is as flat as possible, and the calibration pattern is as clear as possible. A plurality of calibration board images in different poses are shot to make the feature points cover the entire field of view of the camera as much as possible. In this example, a checkerboard calibration board with 14 rows and 20 columns of feature points and a feature point spacing of 60 mm is used, and 17 calibration images in different poses such as image center, directly above, directly below, left, right, upper left, lower left, upper right and lower right are shot.

[0165] The Harris corner point detection algorithm is used to extract the sub-pixel coordinates of the feature points. In this embodiment, the sub-pixel corner point refinement window size is set to (11, 11), the iteration number is 30 times, and the iteration accuracy is 0.001.

[0166] Further, if there are problems such as uneven illumination, camera defocus and excessive image noise in the calibration board image shooting process;

[0167] According to the initial calibration, the re-projection error of each image is calculated, and abnormal images with excessive re-projection error are removed; the calibration board image in the removed image pose is shot, and the camera calibration is performed again.

[0168] Further, the mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system is established, the camera is calibrated according to the mapping relationship, and the initial calibration parameters of the camera are obtained.

[0169] The homography relationship between the world coordinates and the pixel coordinates of the feature points on the shot plurality of images is obtained to establish a projection equation to solve the parameters of the pinhole camera model, and a world coordinate system (O w -Xw Y w Z w ), camera coordinate system (O c -X c Y c Z c ), imaging coordinate system (O-xy) and image coordinate system (O pix -uv). According to the pinhole model, the relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system coordinate m can be expressed as:

[0170]

[0171] In the formula, Z c is a proportional coefficient, f x , f y are the focal lengths of the camera in the imaging coordinate system x, y directions respectively, u0, v0 are the principal point offsets of the camera in the imaging coordinate system x, y directions, K is an intrinsic matrix. R, T are the rotation matrix and translation vector in the extrinsic parameters respectively;

[0172] Further, the camera distortion model is specifically,

[0173]

[0174] In the formula, (x dist , y dist ) is the coordinate of the imaging coordinate system distortion feature point, (x, y) is the coordinate of the imaging coordinate system ideal non-distortion feature point, wherein k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients. The initial solution of the intrinsic and extrinsic parameters is solved by the homography relationship between the feature point world coordinate system coordinate M and the feature point image coordinate system pixel coordinate m. Then the intrinsic and extrinsic parameters and the distortion coefficients solved are globally optimized with the objective of minimizing the re-projection error;

[0175] The expression of the minimized re-projection error is:

[0176]

[0177] In the formula, m ij is the pixel coordinate of the jth (j = 1, ···, m) feature point on the ith (i = 1, ···, n) photographed image, m i ′ j is the pixel coordinate of the re-projected feature point calculated according to the camera geometric model.

[0178] The Levenberg-Marquardt optimization algorithm is used for iterative solution in this embodiment, the number of iterations is set to 500 times, and the iteration convergence accuracy is set to 1e-8.

[0179] The feature point reconstruction module performs a de-distortion operation on the image based on initial calibration of the camera parameters to obtain de-distorted feature point pixel coordinates;

[0180] Further, the detected feature point image coordinate system pixel coordinates m dist (u dist ,v dist ) on the distorted image, the imaging coordinate system coordinates of the feature point are (x dist ,y dist ), and the image coordinate system pixel coordinates m of the feature point on the undistorted image are (u,v), and the imaging coordinate system coordinates are (x,y);

[0181] According to the conversion relationship between the image coordinate system and the imaging coordinate system, the following is obtained:

[0182]

[0183] In the formula, f x ,f y are the focal lengths of the camera in the x and y directions of the imaging coordinate system, and u0 and v0 are the principal point offsets of the camera in the x and y directions of the imaging coordinate system.

[0184] The obtained (x dist ,y dist ) is taken as the initial value (x0,y0),

[0185]

[0186] Through repeated iteration of the distortion model, the de-distorted imaging coordinate system coordinates (x,y) can be obtained,

[0187]

[0188] When the number of iterations or the iteration accuracy requirement is reached, the iteration is exited. The x n ,y n obtained by iteration are taken as the imaging coordinate system coordinates (x,y) on the undistorted image.

[0189] According to the conversion relationship between the image coordinate system and the imaging coordinate system, the following is obtained:

[0190]

[0191] Through the above process, the de-distorted image coordinate system pixel coordinates m of the feature point can be obtained as (u,v);

[0192] The de-distortion model includes a radial distortion model and a tangential distortion model.

[0193] The direction vector of the feature point is obtained based on the pixel coordinates of the feature point, and the included angle of different direction vectors of the feature point is calculated.

[0194] Further, the initial camera internal parameter K obtained through calibration and the pixel coordinates m of the de-distorted image coordinate system are used to perform inverse projection transformation from the image coordinate system to the camera coordinate system to obtain the normalized camera coordinate system coordinates P of the feature point. c ik ,

[0195]

[0196] The normalized camera coordinate system coordinates of the feature point are the optical center O c to the direction vector of the feature point

[0197] The direction vector of the feature point is obtained based on the pixel coordinates of the feature point, and the included angle of different direction vectors of the feature point is calculated. and the unit vector of the optical axis The cosine value cos(α ik ) of the included angle between the direction vector of the feature point and the optical axis is obtained using the cosine formula of the included angle of vectors.

[0198]

[0199] The direction vectors of the kth feature point and the lth feature point of the ith image from the optical center are calculated respectively. The cosine cos(θ kl ) of the included angle between the feature points is obtained according to the cosine formula of the included angle of vectors.

[0200]

[0201] According to the measured distance from the optical center to a certain feature point, the distances from the optical center to all feature points are obtained by using the included angle of the direction vectors of different feature points and the triangular constraint relationship.

[0202] Further, the distance d c from the optical center O ik to a certain feature point M ik is measured using a high-precision distance measuring instrument; in this embodiment, the laser range finder is used to measure the distance from the optical center to the first feature point, which is 2185 mm.

[0203] According to the triangular constraint relationship, the distance d il from the optical center to other feature points M il is calculated; in the triangle ΔO c M ik M il , the length d ik of O c M ik , the distance L between the feature points​kl and the cosine of the included angle between the feature point vectors cos(θ kl ) is known, d il ,

[0204]

[0205] The depth information of each feature point is solved by the distance between each feature point and the optical center, and the world coordinates of all feature points are reconstructed according to the depth information;

[0206] Further, according to the distance d c from the optical center O ik to a certain feature point M ik , the cosine of the included angle between the feature point and the optical axis is obtained; using the projection formula, the depth information Z ik of the feature point M cik is obtained,

[0207] Z cik = d ik cos(α ik ) (12)

[0208] According to the inverse projection formula, the world coordinates M i ′ k of the feature point are reconstructed,

[0209]

[0210] The parameter optimization module optimizes the initial camera calibration parameters to minimize the feature point reconstruction error, and obtains the camera calibration parameters.

[0211] Further, according to the minimum distance between the reconstructed feature point M i ′ j and the real feature point M ij , the internal parameters and distortion parameters are iteratively optimized.

[0212]

[0213] In this embodiment, the Levenberg-Marquardt optimization algorithm is used to optimize the initial camera calibration parameters to minimize the feature point reconstruction error, and the iteration number is set to 200 times.

[0214] The embodiment also includes a camera production line calibration verification system based on monocular three-dimensional reconstruction, the verification system comprising,

[0215] The parameter verification module reconstructs the to-be-tested feature points in the camera coordinate system, calculates the distance between the reconstructed to-be-tested feature points, and compares the distance with the actual distance to obtain the distance measurement error;

[0216] The parameter verification module works by using the distortion correction method and the distortion coefficients obtained by the parameter optimization module to obtain the distortion-free pixel coordinates of the feature point to be tested, and using the camera calibration parameters and distortion coefficients obtained by the parameter optimization module to reconstruct the camera coordinate system coordinates of the feature point to be tested.

[0217] The distance between the feature points to be measured is obtained by using the reconstructed camera coordinate system coordinates, and the relative distance between the feature points to be measured is obtained by comparing it with the actual distance between the feature points to be measured.

[0218] Furthermore, in step 7, the camera coordinate system coordinates of the feature point to be measured are M. c ′ ik M c ′ il ,

[0219]

[0220] The distance L to be measured is obtained using the reconstructed camera coordinate system coordinates. k ′ l ,

[0221] L k ′ l =|M c ′ ik -M c ′ il | (16)

[0222] L k ′ l The actual distance L from the feature point kl The relative distance measurement error ε was obtained by comparison. In this embodiment, the actual distance between the upper left and lower right feature points of the checkerboard calibration board was measured to be 1381.304 mm.

[0223]

[0224] After the distance measurement is completed, output the calibration parameters, reprojection error, and distance measurement relative error.

[0225] Example 3

[0226] This invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any of the steps in Embodiment 1 by running the computer program stored in the memory.

[0227] It should be appreciated that in the embodiments of the present application, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0228] The memory can include read-only memory, flash memory, and random access memory, and provide instructions and data to the processor. Part or all of the memory can also include non-volatile random access memory.

[0229] As can be seen from the above, the electronic device provided by the embodiments of the present application can realize the camera production line calibration method based on monocular three-dimensional reconstruction and the verification method thereof as described in Embodiment One by running a computer program, to solve the problem that the parameter calibration process and the ranging verification experiment process need to be built separately. It is not limited to a specific camera geometric model, and is widely applicable to camera production line calibration verification of various field of view ranges.

[0230] It should be understood that the integrated modules / units described above, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the above-mentioned embodiment methods, which can also be completed by a computer program instructing related hardware. The above-mentioned computer program can be stored in a computer readable storage medium, and when executed by a processor, can realize the steps of the above-mentioned various method embodiments. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The above-mentioned computer readable medium can include any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0231] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0232] It can be clearly understood by a person skilled in the art that, for the convenience and conciseness of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and are not used to limit the protection scope of the application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, and will not be described here.

[0233] It should be noted that the method and its details provided by the above embodiments can be combined into the device and equipment provided by the embodiments, and mutual reference will not be described here.

[0234] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0235] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the above-described device / device embodiments are only schematic, for example, the division of the above modules or units is only a logical function division, and actual implementation can be different, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0236] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A camera production line calibration method based on monocular three-dimensional reconstruction, characterized in that, The method comprises the following steps, Step 1: Obtain initial camera calibration parameters by using Zhang Zhengyou calibration method; Step 2: Measure the distance from the camera optical center to a certain feature point, obtain the distance from the camera optical center to all feature points according to geometric constraint relationship, and reconstruct the world coordinates of all feature points; The step 2 specifically comprises the following steps, Step 2.1: Perform de-distortion operation on the image based on the initial camera calibration parameters to obtain the de-distorted feature point pixel coordinates; Step 2.2: Obtain the feature point direction vector based on the feature point pixel coordinates, and calculate the included angle of different feature point direction vectors; Step 2.3: Measure the distance from the optical center to a certain feature point, and obtain the distance from the optical center to all feature points by using the included angle of the feature point direction vector and the triangle constraint relationship; Step 2.4: Solve the depth information of each feature point, and reconstruct the world coordinates of all feature points according to the depth information; Step 3: Optimize the initial camera calibration parameters to minimize the feature point reconstruction error, and obtain the camera calibration parameters.

2. The camera production line calibration method based on monocular three-dimensional reconstruction according to claim 1, characterized in that, The step 1 specifically comprises the following steps, Step 1.1: Capture multiple calibration board images by using the camera to be calibrated, and obtain the pixel coordinates of the feature points; Step 1.2: Establish the mapping relationship between the pixel coordinates of the feature point image coordinate system and the coordinates of the feature point world coordinate system, calibrate the camera according to the mapping relationship, and obtain the initial calibration parameters of the camera.

3. The camera production line calibration method based on monocular three-dimensional reconstruction according to claim 1, characterized in that, The step 2.1 is specifically, setting the feature point image coordinate system pixel coordinates detected on the distorted image For The imaging coordinate system coordinates of the feature point are On the non-distorted image, the image coordinate system pixel coordinates corresponding to the feature point are For The imaging coordinate system coordinates are ; The de-distortion operation is performed through a de-distortion model to obtain de-distorted image coordinate system pixel coordinates To ; The de-distortion model comprises a radial distortion model and a tangential distortion model.

4. The camera production line calibration method based on monocular three-dimensional reconstruction of claim 1, wherein, The step 2.2 is specifically, according to the camera initial internal parameter obtained by calibration and the pixel coordinates of the undistorted image coordinate system performing inverse projection transformation from the image coordinate system to the camera coordinate system to obtain the normalized camera coordinate system coordinates of the feature points ; The normalized camera coordinate system coordinate of the feature point is the optical center The direction vector to the feature point ; According to the direction vector of the feature point Unit vector of the optical axis The cosine value of the angle between the direction vector of the feature point and the optical axis is obtained using the vector included angle cosine formula ; The first of the respective images is calculated The first of the feature points is calculated , ; According to the cosine formula of the vector angle, the cosine of the angle between the characteristic points is obtained .

5. The camera production line calibration method based on monocular three-dimensional reconstruction of claim 1, wherein, The step 2.3 is specifically using high-precision ranging instrument to measure the optical center to a feature point distance ; The distance from the optical center to other feature points is calculated based on the triangle constraint relationship. distance ; in triangle middle, length Distance between feature points and the cosine of the angle between feature point vectors It is known that the solution can be found using the Law of Cosines. .

6. The camera production line calibration method based on monocular three-dimensional reconstruction according to claim 1, characterized in that, The step 2.4 is specifically, according to the light center to a feature point The distance is The cosine of the angle between the obtained feature point and the optical axis; The characteristic points can be obtained using a projection formula depth information The characteristic points can be reconstructed according to an inverse projection formula .

7. The camera production line calibration method based on monocular three-dimensional reconstruction according to claim 1, characterized in that, The step 3 is specifically, according to the reconstructed feature points The distance between the real feature points The internal parameters and distortion parameters are continuously iteratively optimized with the target of minimum distance. 8.A method for verifying camera production line calibration based on monocular three-dimensional reconstruction, characterized in that, The verification method comprises using the monocular three-dimensional reconstruction-based camera production line calibration method according to any one of claims 1-7, and the verification method specifically comprises, After obtaining the camera calibration parameters, reconstruct the to-be-measured feature points in the camera coordinate system, calculate the distance between the reconstructed to-be-measured feature points, and compare the distance with the actual distance to obtain the distance measurement error.

9. The verification method of camera production line calibration based on monocular three-dimensional reconstruction according to claim 8, characterized in that, Re-capture a group of calibration board images, obtain the to-be-measured feature point non-distortion pixel coordinates by using the de-distortion method and the obtained distortion coefficients, and reconstruct the camera coordinate system coordinates of the to-be-measured feature points by using the obtained camera calibration parameters and distortion coefficients; Obtain the distance between the to-be-measured feature points by using the reconstructed camera coordinate system coordinates, compare the distance with the actual distance of the to-be-measured feature points to obtain the distance measurement relative error.

10. The verification method of camera production line calibration based on monocular three-dimensional reconstruction according to claim 9, characterized in that, The camera coordinate system coordinates of the to-be-tested feature point are , , （15） The camera coordinate system coordinates are obtained by reconstruction , （16） Will Actual distance from feature point Comparison of ranging relative error , （17） Output the calibration parameters, the re-projection error, and the distance measurement relative error after the distance measurement is completed.

11. A monocular three-dimensional reconstruction based camera production line calibration system, characterized in that, The system uses the monocular three-dimensional reconstruction-based camera production line calibration method according to any one of claims 1-7, and the system comprises, A camera calibration module that obtains initial calibration parameters of the camera by using Zhang Zhengyou calibration method; A feature point reconstruction module that performs de-distortion operation on the image based on the initial camera calibration parameters to obtain de-distorted feature point pixel coordinates; Obtain the feature point direction vector based on the feature point pixel coordinates, and calculate the included angle of different feature point direction vectors; Obtain the distance from the optical center to all feature points by using the included angle of the feature point direction vector and the triangle constraint relationship according to the measured distance from the optical center to a certain feature point; Solve the depth information of each feature point, and reconstruct the world coordinates of the feature points according to the depth information; A parameter optimization module that optimizes the initial camera calibration parameters to minimize the feature point reconstruction error, and obtains the camera calibration parameters.

12. A verification system for camera production line calibration based on monocular three-dimensional reconstruction, characterized in that, The verification system comprises the monocular three-dimensional reconstruction-based camera production line calibration system according to claim 11, and the verification method for the monocular three-dimensional reconstruction-based camera production line calibration according to claim 8, and the system comprises, The parameter verification module reconstructs the to-be-measured feature points in the camera coordinate system, calculates the distances between the to-be-measured feature points, and compares the distances with the actual distances to obtain ranging errors; The working process of the parameter verification module is specifically as follows: the distortion-free pixel coordinates of the to-be-measured feature points are obtained by using the distortion coefficients obtained by the distortion removal method and the parameter optimization module, the camera coordinate system coordinates of the to-be-measured feature points are reconstructed by using the camera calibration parameters and the distortion coefficients obtained by the parameter optimization module; The distances between the to-be-measured feature points are obtained by using the reconstructed camera coordinate system coordinates, and the ranging relative errors are obtained by comparing the distances with the actual distances of the to-be-measured feature points.

13. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1-7 or the method according to claim 8.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1-7 or the method according to claim 8.

Images