Parameter calibration method, device, equipment and storage medium for binocular fisheye camera

Abstract

The present invention relates to the field of image processing technology, and in particular to a parameter calibration method, apparatus, device, and storage medium for a binocular fisheye camera. The method labels a calibration plate, extracts the calibration plate image, and detects landmarks using a deep learning approach. Standard data for the calibration plate is obtained by sorting the landmarks and performing corrections on the sorted labeled points. The calibration plate image undergoes dedistortion correction and backprojection to obtain a correspondence between the coordinates of the calibration plate image at the front view angle and the coordinates of the image to be measured, thereby eliminating the effect of fisheye distortion on the accuracy of target extraction from the calibration plate. The relative positional relationship of the binocular cameras is obtained using an external parameter calibration field, and then the binocular cameras are calibrated using the calibration intrinsic parameters of each camera. Dedistortion correction improves the accuracy of the binocular camera's intrinsic calibration, solving the current technical problem of low parameter calibration accuracy for cameras with large distortion.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a parameter calibration method, device, equipment and storage medium for a binocular fisheye camera. Background Art

[0002] Optical camera calibration is a fundamental task in computer vision, and its main task is to recover the intrinsic and extrinsic parameters of the target camera.

[0003] Current optical camera calibration toolboxes only offer an intrinsic parameter calibration toolbox, but lack an extrinsic parameter calibration toolbox. Furthermore, during intrinsic parameter calibration, the distortion of the calibration plate can lead to inaccurate landmark extraction for cameras with large distortion, such as wide-angle and fisheye cameras, affecting calibration accuracy. Therefore, improving the calibration accuracy of cameras with large distortion has become a pressing technical challenge. Summary of the Invention

[0004] The main purpose of the present invention is to provide a parameter calibration method, device, equipment and storage medium for a binocular fisheye camera, aiming to solve the technical problem of low parameter calibration accuracy of existing large distortion cameras.

[0005] To achieve the above object, the present invention provides a parameter calibration method for a binocular fisheye camera, the parameter calibration method for a binocular fisheye camera comprising:

[0006] Obtain at least two images to be tested captured by a binocular camera, extract the calibration plate and landmarks in the images to be tested based on a pre-trained network and a deep learning method, and obtain a calibration plate image and target landmarks;

[0007] Determining standard parameters of the calibration plate image based on a result of sorting the target landmark points;

[0008] Correcting the calibration plate image based on a dedistortion formula to obtain a corrected calibration plate, and projecting the corrected calibration plate onto the image to be measured based on a preset projection algorithm to obtain a projected calibration plate;

[0009] Based on the projection calibration plate and the standard parameters, the image to be measured is calibrated to obtain the target internal parameters of the binocular camera, and the target internal parameters are iteratively optimized based on an iterative optimization method to obtain calibration internal parameters;

[0010] Based on the calibration plate extracted from at least two test images captured by the binocular camera, an extrinsic calibration field is obtained, and based on the extrinsic calibration field and the calibration intrinsic parameters, the calibration extrinsic parameters of the binocular camera are calculated.

[0011] Furthermore, before correcting the calibration plate image based on the dedistortion formula to obtain the corrected calibration plate, the method further includes:

[0012] Based on the fisheye camera model, the calibration plate is calibrated to obtain an initial intrinsic parameter matrix and initial distortion parameters;

[0013] The dedistortion formula is generated based on the initial internal parameter matrix and the initial distortion parameter.

[0014] Furthermore, the calibration plate image is corrected based on the dedistortion formula to obtain a corrected calibration plate, including:

[0015] Based on the calibration plate image, obtaining a calibration plate convex hull;

[0016] Based on the dedistortion formula, dedistortion calculation is performed on the convex hull of the calibration plate to obtain a dedistorted convex hull;

[0017] Based on the homography matrix, the dedistorted convex hull is mapped to obtain the correction calibration plate.

[0018] Furthermore, the step of correcting the calibration plate image based on the dedistortion formula to obtain a corrected calibration plate further includes:

[0019] Based on the fisheye camera model, projection calculation is performed on the calibration plate to obtain a current intrinsic parameter matrix and current distortion parameters;

[0020] Based on the current intrinsic parameter matrix and the current distortion parameter, a current dedistortion formula is generated, and based on the current dedistortion formula, dedistortion correction is performed on the calibration plate to obtain a current calibration plate;

[0021] Based on the fisheye camera model, performing projection calculation on the current calibration plate to generate an iterative intrinsic parameter matrix and iterative distortion parameters, and performing intrinsic parameter calibration of the fisheye camera based on the iterative intrinsic parameter matrix, the iterative distortion parameters, and an iterative optimization method to obtain current calibration internal parameters;

[0022] Based on the iterative internal parameter matrix and the iterative distortion parameter, an iterative dedistortion formula is generated, the current calibration plate is dedistorted and corrected, and an iterative calibration plate is generated and used as the current calibration plate until the calibration accuracy of the current calibration plate reaches a preset accuracy. The current calibration internal parameter is used as the calibration internal parameter, and the current calibration plate is used as the correction calibration plate.

[0023] Furthermore, the calculating of the calibration extrinsic parameters of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameters includes:

[0024] Calculating a relative pose estimate of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameter;

[0025] Based on a preset iterative optimization algorithm, the relative pose estimation value is iteratively optimized to obtain the target pose as the calibration extrinsic parameter of the binocular camera.

[0026] Furthermore, the iterative optimization of the relative pose estimation value based on a preset iterative optimization algorithm to obtain the target pose as the calibration extrinsic parameter of the binocular camera includes:

[0027] Based on a coordinate transformation algorithm, the coordinate reference point in the external parameter calibration field is transformed into the image to be measured to obtain the observed coordinates of the coordinate reference point;

[0028] Calculating the calculated coordinates of the coordinate reference point based on the calibration internal parameters, and determining the current posture change based on the observed coordinates and the calculated coordinates;

[0029] Adjusting the calibration extrinsic parameter based on the current posture change to obtain the current posture as the calibration extrinsic parameter;

[0030] Based on the coordinate transformation algorithm, the coordinate reference point is transformed into the image to be measured to obtain the observed coordinates, and based on the observed coordinates and the calculated coordinates, the current posture change is calculated until the current posture change reaches a preset range. Based on the current posture change, the calibration external parameter is adjusted to obtain the current posture as the calibration external parameter.

[0031] Furthermore, the method of extracting the calibration plate and landmarks in the image to be tested based on the pre-trained network and the deep learning method to obtain the calibration plate image and target landmarks includes:

[0032] Based on a preset annotation method, data annotation is performed on the image to be tested to obtain calibration plate data and landmark point data;

[0033] Based on the pre-trained network and the deep learning method, the calibration plate data and the marker point data are trained to extract the calibration plate image and the target marker points in the calibration plate image.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a parameter calibration device for a binocular fisheye camera, the parameter calibration device for a binocular fisheye camera comprising:

[0035] A calibration plate extraction module is used to obtain at least two test images captured by the binocular camera, and extract the calibration plate and landmarks in the test images based on a pre-trained network and a deep learning method to obtain a calibration plate image and target landmarks;

[0036] A marker point sorting module, configured to determine standard parameters of the calibration plate image based on a sorting result of the target marker points;

[0037] A calibration plate correction and projection module is used to correct the calibration plate image based on a dedistortion formula to obtain a corrected calibration plate, and project the corrected calibration plate onto the image to be measured based on a preset projection algorithm to obtain a projected calibration plate;

[0038] An internal parameter calibration module is used to calibrate the image to be measured based on the projection calibration plate and the standard parameters to obtain the target internal parameters of the binocular camera, and iteratively optimize the target internal parameters based on an iterative optimization method to obtain calibration internal parameters;

[0039] An extrinsic parameter calibration module is used to obtain an extrinsic parameter calibration field based on the calibration plate extracted from at least two test images captured by the binocular camera, and calculate the calibration extrinsic parameters of the binocular camera based on the extrinsic parameter calibration field and the calibration intrinsic parameters.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a parameter calibration device for a binocular fisheye camera, wherein the parameter calibration device for a binocular fisheye camera includes a processor, a memory, and a parameter calibration program for a binocular fisheye camera stored in the memory and executable by the processor, wherein when the parameter calibration program for a binocular fisheye camera is executed by the processor, the steps of the parameter calibration method for a binocular fisheye camera as described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a parameter calibration program for a binocular fisheye camera is stored. When the parameter calibration program for a binocular fisheye camera is executed by a processor, the steps of the parameter calibration method for a binocular fisheye camera as described above are implemented.

[0042] The present invention provides a parameter calibration method for a binocular fisheye camera. The method obtains at least two test images captured by a binocular camera, extracts a calibration plate and landmarks from the test images based on a pretrained network and a deep learning method, and obtains a calibration plate image and target landmarks; determines standard parameters of the calibration plate image based on a sorting result of the target landmarks; corrects the calibration plate image based on a dedistortion formula to obtain a corrected calibration plate, and projects the corrected calibration plate onto the test images based on a preset projection algorithm to obtain a projected calibration plate; calibrates the test images based on the projected calibration plate and the standard parameters to obtain target intrinsic parameters of the binocular camera, and iteratively optimizes the target intrinsic parameters based on an iterative optimization method to obtain calibration intrinsic parameters; obtains an extrinsic parameter calibration field based on the calibration plate extracted from the at least two test images captured by the binocular camera, and calculates calibration extrinsic parameters of the binocular camera based on the extrinsic parameter calibration field and the calibration intrinsic parameters. Through the above method, by dedistorting and back-projecting the calibration plate image, the coordinate correspondence between the calibration plate image in the front view angle and the coordinate correspondence between the coordinates of the image to be measured and the calibration points is obtained by sorting the calibration points, and then the standard parameters of the calibration plate are calculated for the calibration plate parameters in the image to be measured. The relative position relationship between the calibration plate and the binocular camera is obtained by the external parameter calibration field, and then the calibration external parameters of the binocular camera are calculated using the calibration internal parameters of each camera, thereby achieving the internal and external parameter calibration of the binocular camera. Through dedistortion correction, the accuracy of coordinate correspondence can be improved, thereby improving the parameter calculation accuracy of the distorted image, and then improving the internal parameter calibration accuracy of the binocular camera, solving the current technical problem of low parameter calibration accuracy of large distortion cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the hardware structure of a binocular fisheye camera parameter calibration device involved in an embodiment of the present invention;

[0044] Figure 2 1 is a flow chart of a first embodiment of a method for calibrating parameters of a binocular fisheye camera according to the present invention;

[0045] Figure 3 1 is a flow chart of a second embodiment of a method for calibrating parameters of a binocular fisheye camera according to the present invention;

[0046] Figure 4 1 is a flow chart of a third embodiment of a method for calibrating parameters of a binocular fisheye camera according to the present invention;

[0047] Figure 5 1 is a flow chart of a fourth embodiment of a method for calibrating parameters of a binocular fisheye camera according to the present invention;

[0048] Figure 6Schematic diagram of the functional modules of the first embodiment of the parameter calibration device for a binocular fisheye camera of the present invention.

[0049] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] The binocular fisheye camera parameter calibration method involved in the embodiment of the present invention is mainly applied to a binocular fisheye camera parameter calibration device, which can be a device with display and processing functions such as a PC, a portable computer, or a mobile terminal.

[0052] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the hardware structure of the parameter calibration device for a binocular fisheye camera involved in the embodiment of the present invention. In the embodiment of the present invention, the parameter calibration device for a binocular fisheye camera may include a processor 1001 (e.g., a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components; the user interface 1003 may include a display screen and an input unit such as a keyboard; the network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., a WI-FI interface); and the memory 1005 may be a high-speed RAM memory or a non-volatile memory such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation on the parameter calibration device of the binocular fisheye camera, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, and a parameter calibration program for a binocular fisheye camera.

[0055] exist Figure 1In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the parameter calibration program of the binocular fisheye camera stored in the memory 1005 and execute the parameter calibration method of the binocular fisheye camera provided in the embodiment of the present invention.

[0056] An embodiment of the present invention provides a parameter calibration method for a binocular fisheye camera.

[0057] Reference Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a parameter calibration method for a binocular fisheye camera according to the present invention.

[0058] In this embodiment, the parameter calibration method of the binocular fisheye camera includes the following steps:

[0059] Step S10: obtaining at least two pictures to be tested captured by a binocular camera, extracting a calibration plate and landmarks from the pictures to be tested based on a pre-trained network and a deep learning method, and obtaining a calibration plate image and target landmarks;

[0060] Step S20, determining standard parameters of the calibration plate image based on the sorting result of the target landmark points;

[0061] In this embodiment, at least two images captured synchronously by each of the binocular cameras are used as test images. Each test image corresponds to a camera and is captured synchronously. The scene captured by the test image may be large, and the calibration plate may occupy a relatively small proportion of the space in the test image, so the calibration plate needs to be extracted and cropped.

[0062] In an exemplary embodiment, assuming each image contains only one calibration plate, the calibration plate occupies a limited area of ​​the image during acquisition. To facilitate subsequent processing, the calibration plate must first be processed and extracted. During extraction, a pretrained network is used to identify the calibration plate portion and crop it from the original image. The pretrained network can be an object detection network, which uses LabelImage software to annotate the target calibration plate and landmarks. The calibration plate and landmarks can be annotated separately, and then the neural network can be trained separately.

[0063] Among them, the target detection network YOLOv5 and the feature filtering network VGG16 can be used to train the calibration plate, and the calibration plate can be cropped out from the test picture to obtain the calibration plate image; the target detection network YOLOv5 that tends to detect small objects can be used to train the landmark points, and the anchor points and detection boxes in the YOLOv5 model can be adjusted to 5%-10% of the size of the landmark plate. In this process, YOLOv5 tends to efficiently extract landmark points on the cropped calibration plate.

[0064] Step S30, correcting the calibration plate image based on a dedistortion formula to obtain a corrected calibration plate, and projecting the corrected calibration plate onto the image to be measured based on a preset projection algorithm to obtain a projected calibration plate;

[0065] In this embodiment, the maximum convex hull of the calibration plate is obtained by sorting the initially extracted landmarks. The extracted and cropped calibration plate is first dedistorted and converted to an orthographic projection. The calibration plate is dedistorted using the Kanala-Brandt model and a rough intrinsic parameter matrix and distortion parameters obtained using traditional methods. After obtaining the dedistorted convex hull using the formula, the homography matrix H can be estimated using RANSAC, which maps it to a rectangle in a virtual coordinate system. Dedistortion and homography transformation can be combined into atomic operations, so that the orthographic calibration plate can be sampled from the original calibration plate without sampling loss. After the orthographic calibration plate is obtained, the corner points on the calibration plate can be accurately extracted.

[0066] In an exemplary embodiment, the preset projection algorithm may be a back-projection algorithm, in which each corner point extracted from the calibration plate of the orthographic view is back-projected according to the existing initial parameters to obtain the original corner point, and represented on the fisheye image.

[0067] Step S40: calibrating the image to be measured based on the projection calibration plate and the standard parameters to obtain the target internal parameters of the binocular camera, and iteratively optimizing the target internal parameters based on an iterative optimization method to obtain calibration internal parameters;

[0068] In this embodiment, the internal reference is obtained by using Zhang Zhengyou's internal reference calibration method in the picture to be tested. Among them, Zhang Zhengyou's internal reference calibration method uses a calibration plate composed of two-dimensional squares for calibration, collects pictures of the calibration plate in different postures, extracts the pixel coordinates of the corner points in the picture, calculates the initial values ​​of the internal and external parameters of the camera through the homography matrix, estimates the distortion coefficient using the nonlinear least squares method, and finally optimizes the parameters using the maximum likelihood estimation method. This method is between the photographic calibration method and the self-calibration method. It not only overcomes the shortcomings of the high-precision three-dimensional calibration object required by the photographic calibration method, but also solves the problem of poor robustness of the self-calibration method. The calibration process only requires the use of a printed chessboard and taking several sets of pictures from different directions. Anyone can make their own calibration pattern. It is not only practical, flexible and convenient, but also has high accuracy and good robustness.

[0069] Step S50: obtaining an extrinsic calibration field based on the calibration plate extracted from at least two test images captured by the binocular camera, and calculating the calibration extrinsic parameters of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameters.

[0070] In this embodiment, for a binocular fisheye camera, the extrinsic parameters of each camera can be estimated individually using the PnP algorithm. The relative rotation and translation can be calculated using formulas, and the baselines between different viewpoints can also be calculated. Accuracy assessment can be easily performed using the rotation matrix and baselines.

[0071] Specifically, the initial value calculation of the extrinsic parameters adopts a method to calculate the relative pose of the camera under the control field. For a calibration control field with the same coordinate system, the PnP algorithm is used to calculate the initial pose of each camera relative to the control field as the calibration extrinsic parameter of the camera.

[0072] This embodiment provides a parameter calibration method for a binocular fisheye camera. The method extracts a calibration plate image from a test image through a pre-trained network, thereby facilitating recognition and dedistortion of the calibration plate. The calibration plate image is dedistorted and corrected to obtain a calibrated calibration plate with an orthographic viewing angle. The calibration points on the calibration plate are sorted to obtain standard parameters of the calibration plate, which serve as a reference for parameter calibration. The calibration plate is then reverse-projected onto the test image through a preset projection algorithm, thereby projecting the corner points on the calibration plate onto the test image. The test image is calibrated based on the standard parameters and the projection coordinates of the calibration plate on the test image, thereby obtaining the calibration intrinsic parameters of the binocular camera. The camera extrinsic parameters are calculated based on the calibration intrinsic parameters and an extrinsic calibration field formed by the test image corresponding to the binocular camera, thereby obtaining the calibration extrinsic parameters of each camera. Through the above method, by dedistorting and back-projecting the calibration plate image, the coordinate correspondence between the calibration plate image in the front view angle and the coordinate correspondence between the coordinates of the image to be measured and the coordinates of the image to be measured is obtained. By sorting the calibration points, the standard parameters of the calibration plate are obtained, and then the parameters of the calibration plate in the image to be measured are calculated. The dedistortion correction can improve the accuracy of the coordinate correspondence, thereby improving the accuracy of the parameter calculation of the distorted image, and further improving the intrinsic parameter calibration accuracy of the binocular camera. At the same time, the relative position relationship between the calibration plate and the binocular camera is obtained using the external parameter calibration field, and then the calibration extrinsic parameters of the binocular camera are calculated using the calibration intrinsic parameters of each camera, thereby achieving the internal and external parameter calibration of the binocular camera. This solves the technical problem of low parameter calibration accuracy of large distortion cameras.

[0073] Reference Figure 3 , Figure 3 FIG. 4 is a flow chart of a second embodiment of a parameter calibration method for a binocular fisheye camera according to the present invention.

[0074] Based on the above Figure 2 In the embodiment shown, in this embodiment, before step S30, the following steps are further included:

[0075] Step S01: calibrating the calibration plate based on a fisheye camera model to obtain an initial intrinsic parameter matrix and initial distortion parameters;

[0076] Step S02: generating the dedistortion formula based on the initial intrinsic parameter matrix and the initial distortion parameter.

[0077] For the calibration plate extracted and cut out, the rough initial internal parameter matrix and initial distortion parameters are obtained by the traditional method through the Kannala-Brandt model, which are defined as M0, Calibration plate and its convex hull The distortion is removed by the following process. For each point P, the distortion is removed by the following formula:

[0078]

[0079] in, p' is defined as the point after dedistortion by the dedistortion mapping and perspective projection.

[0080] The Kannala-Brandt model is a universal fisheye camera model. Fisheye lenses cannot be precisely designed according to the projection model during production. Therefore, to facilitate fisheye camera calibration, Kannala-Brandt proposed a general polynomial approximation model for fisheye cameras. This model is applicable to standard, wide-angle, and fisheye lenses. The Kannala-Brandt model assumes a proportional relationship between the distance from the image optical center to the projection point and the polynomial angle between the projection ray and the principal axis at that point. Based on this proportional relationship, distorted points in the fisheye image can be calculated from points in the undistorted image.

[0081] Furthermore, based on the above Figure 3 In the illustrated embodiment, step S30 specifically includes:

[0082] Based on the calibration plate image, obtaining a calibration plate convex hull;

[0083] Based on the dedistortion formula, dedistortion calculation is performed on the convex hull of the calibration plate to obtain a dedistorted convex hull;

[0084] Based on the homography matrix, the dedistorted convex hull is mapped to obtain the correction calibration plate.

[0085] In this embodiment, correction is completed under the fisheye perspective through dedistortion. Although the calibration plate has been dedistorted and can theoretically be processed like a small-distortion image, some points at the edge of the image are still difficult to detect and process due to excessive stretching. Therefore, the perspective projection can be converted into an orthographic image.

[0086] The dedistorted convex hull is obtained by the formula After that, RANSAC can be used to estimate the homography matrix H, and the Mapped to rectangle Dedistortion and homography transformation can be combined into an atomic operation, so the orthographic calibration plate can be sampled from the original calibration plate without sampling loss.

[0087] Among them, RANSAC mainly solves the problem of outliers in samples and can handle up to 50% of outlier cases. RANSAC achieves its goal by repeatedly selecting a set of random subsets from the data. The selected subsets are assumed to be inliers and verified using the following method: there is a model that applies to the assumed inliers, that is, all unknown parameters can be calculated from the assumed inliers. This model is used to test all other data. If a point applies to the estimated model, it is considered to be an inlier. If enough points are classified as assumed inliers, then the estimated model is reasonable enough. Then, all the assumed inliers are used to re-estimate the model, because it has only been estimated by the initial assumed inliers. Finally, the model is evaluated by estimating the error rate between the inliers and the model.

[0088] A homography describes the mapping relationship between points on the same plane in different images. A homography is a concept in projective geometry, also known as a projective transformation. It maps points (three-dimensional homogeneous vectors) on one projective plane to another, and maps straight lines to straight lines, preserving the line. When the camera is rotated but not translated, the homography matrix H can be used to describe the correspondence between the points in the two images.

[0089] Furthermore, based on the above Figure 3 In the illustrated embodiment, step S30 specifically further includes:

[0090] Based on the fisheye camera model, projection calculation is performed on the calibration plate to obtain a current intrinsic parameter matrix and current distortion parameters;

[0091] Based on the current intrinsic parameter matrix and the current distortion parameter, a current dedistortion formula is generated, and based on the current dedistortion formula, dedistortion correction is performed on the calibration plate to obtain a current calibration plate;

[0092] Based on the fisheye camera model, performing projection calculation on the current calibration plate to generate an iterative intrinsic parameter matrix and iterative distortion parameters, and performing intrinsic parameter calibration of the fisheye camera based on the iterative intrinsic parameter matrix, the iterative distortion parameters, and an iterative optimization method to obtain current calibration internal parameters;

[0093] Based on the iterative internal parameter matrix and the iterative distortion parameter, an iterative dedistortion formula is generated, the current calibration plate is dedistorted and corrected, and an iterative calibration plate is generated and used as the current calibration plate until the calibration accuracy of the current calibration plate reaches a preset accuracy. The current calibration internal parameter is used as the calibration internal parameter, and the current calibration plate is used as the correction calibration plate.

[0094] In this embodiment, after processing the fisheye image and obtaining relatively accurate corner points, in order to obtain more accurate internal parameters of the wide-angle camera or fisheye camera, a rough internal parameter matrix and distortion parameter M0 are obtained. Afterwards, it is used to estimate a more accurate intrinsic parameter matrix and distortion parameters, and further estimate a more accurate orthographic viewing angle, extract more accurate corner points, and estimate new intrinsic parameter matrix and distortion parameters. This process is repeated until the intrinsic parameter matrix and distortion parameters that meet the preset accuracy standards are obtained. The high-precision calibration internal parameters and calibration plate are calculated using the intrinsic parameter matrix and distortion parameters.

[0095] Reference Figure 4 , Figure 4 FIG. 4 is a flow chart of a third embodiment of a method for calibrating parameters of a binocular fisheye camera according to the present invention.

[0096] Based on the above Figure 2 In the embodiment shown, in this embodiment, step S50 specifically includes:

[0097] Step S51: Calculating the relative pose estimation value of the binocular camera based on the external parameter calibration field and the calibration internal parameter;

[0098] Step S52: Based on a preset iterative optimization algorithm, iteratively optimize the relative pose estimation value to obtain the target pose as the calibration extrinsic parameter of the binocular camera.

[0099] In this embodiment, the initial value calculation of the extrinsic parameters adopts a method for calculating the relative pose of the camera under the control field. For a calibration control field with the same coordinate system, the PnP algorithm is used to calculate the initial pose of each camera relative to the control field. For camera extrinsic parameter calibration, the extrinsic parameters of each camera are estimated separately using the PnP algorithm. The relative rotation and translation can be obtained by calculation, so the baseline of the binocular camera can also be obtained. Then, an iterative optimization method is used to optimize the internal and external parameters of the fisheye camera: by calculating the Jacobian matrix of the fisheye camera, the distortion coefficient of the fisheye camera is introduced into the optimization method, and then optimization is performed using the bundle adjustment method to obtain the optimized parameters.

[0100] Specifically, under normal circumstances, the world coordinate system and the camera coordinate system do not coincide. In this case, when a point in the world coordinate system is to be projected onto the image plane, the coordinates of the point must first be converted to the camera coordinate system. The process of converting a rigid body from the world coordinate system to the camera coordinate system can be obtained through rotation and translation. PnP (Perspective-n-Point) is a method for solving the motion of 3D to 2D point pairs, with the goal of solving the pose of the camera coordinate system relative to the world coordinate system. It describes how to estimate the pose of the camera (i.e., solving the rotation matrix R and translation vector t from the world coordinate system to the camera coordinate system) when the coordinates of n 3D points (relative to the world coordinate system) and the pixel coordinates of these points are known.

[0101] Among them, the mathematical formula for camera pose estimation can be expressed as:

[0102]

[0103] Among them, p is the coordinate of the point in the pixel coordinate system, P C is the coordinate of the point in the camera coordinate system, P W is the coordinate of the point in the world coordinate system, ω is the depth of the point, K is the intrinsic parameter matrix of the camera, R CW and is the pose transformation from the world coordinate system to the camera coordinate system.

[0104] It is understood that there are many ways to calculate the relative pose of a camera, including linear methods such as direct linear transformation (DLT), P3P, and EPnP, as well as nonlinear methods such as bundle adjustment (BA). This application uses one of these methods as an example to illustrate the specific distance, but this does not mean that only this method can be used to implement the extrinsic parameter calibration described in the embodiments of this application.

[0105] In a specific embodiment, the PnP problem can be formulated as a nonlinear least-squares problem involving reprojection error. Linear methods often treat the positions of spatial points as known variables and construct a system of equations to solve the camera pose. Nonlinear optimization, on the other hand, treats both the camera pose and the spatial point positions as optimization variables and optimizes them together. This method combines the camera pose and the 3D point positions to minimize the reprojection error, resulting in a holistic optimization approach.

[0106] In an exemplary embodiment, there is a deviation between the projected coordinates in the pixel coordinate system obtained by observation and the projected coordinates in the pixel coordinate system calculated by the camera pose and the 3D point position, which is called the reprojection error. Generally speaking, non-homogeneous coordinates are used when calculating the error, that is, the errors in the two axis directions of the pixel coordinate system, so the reprojection error is only two-dimensional. The estimated value of the camera pose can be adjusted to reduce the distance between the point coordinates in the pixel coordinate system and the point coordinates in the world coordinate system. In the process of adjusting the estimated value of the camera pose to reduce the overall error, it is most important to know the derivative of the error with respect to the camera pose. Only by knowing the derivative can the direction of adjustment of the estimated value of the camera pose be determined. The derivative is the derivative of the error term with respect to the optimization variable, that is, linearization.

[0107] Furthermore, based on the above Figure 2 In the embodiment shown, in this embodiment, step S52 specifically includes:

[0108] Based on a coordinate transformation algorithm, the coordinate reference point in the external parameter calibration field is transformed into the image to be measured to obtain the observed coordinates of the coordinate reference point;

[0109] Calculating the calculated coordinates of the coordinate reference point based on the calibration internal parameters, and determining the current posture change based on the observed coordinates and the calculated coordinates;

[0110] Adjusting the calibration extrinsic parameter based on the current posture change to obtain the current posture as the calibration extrinsic parameter;

[0111] Based on the coordinate transformation algorithm, the coordinate reference point is transformed into the image to be measured to obtain the observed coordinates, and based on the observed coordinates and the calculated coordinates, the current posture change is calculated until the current posture change reaches a preset range. Based on the current posture change, the calibration external parameter is adjusted to obtain the current posture as the calibration external parameter.

[0112] In this embodiment, after obtaining the camera pose, an iterative optimization method is used to optimize the internal and external parameters of the fisheye camera. By calculating the Jacobian matrix of the fisheye camera, the distortion coefficients of the fisheye camera are introduced into the optimization method, and then optimization is performed using the bundle adjustment method to obtain the optimized parameters. The Jacobian matrix represents the first-order variation relationship of the reprojection error with respect to the Lie algebra of the camera pose, thereby optimizing the camera pose and reducing the reprojection error. The derivative matrix of the reprojection error vector with respect to the camera pose and the position of the feature points is then calculated. During the process of optimizing and iteratively adjusting the camera pose estimate, this derivative matrix provides the gradient direction to guide the optimization iteration.

[0113] In one exemplary embodiment, during each iteration, for each spatial point, the coordinates of the spatial point in the world coordinate system are first transformed to the camera coordinate system. The pixel coordinates are then calculated using the camera intrinsic parameters. The observed pixel coordinates are then subtracted from the calculated values ​​to obtain an error vector and a Jacobian matrix. After calculating the error vector and Jacobian matrix for each spatial point, a system of linear equations is obtained. Solving this system of linear equations calculates the change in camera pose. The current camera pose is then adjusted based on this change, achieving iteration. This iteration continues until the change in camera pose is sufficiently small, resulting in a high-precision camera pose that serves as the camera extrinsic parameter.

[0114] Reference Figure 5 , Figure 5 FIG4 is a flow chart of a fourth embodiment of a method for calibrating parameters of a binocular fisheye camera according to the present invention.

[0115] Based on the above Figure 2 In the embodiment shown, in this embodiment, step S10 specifically includes:

[0116] Step S11, based on a preset annotation method, performing data annotation on the image to be tested to obtain calibration plate data and landmark point data;

[0117] Step S12: Based on the pre-trained network and the deep learning method, the calibration plate data and the marker point data are trained to extract the calibration plate image and the target marker points in the calibration plate image.

[0118] In this embodiment, a calibration detection method based on deep learning detects the calibration plate and landmarks in the image to be tested, and crops the calibration plate in the image to be tested to obtain a calibration plate image.

[0119] Specifically, a method based on combining an object detection network (YOLOv5) and a feature filtering network (VGG16) is used to obtain a highly accurate calibration plate and landmarks. The target calibration plate and landmarks are first labeled using LabelImage software, where the labeling is done separately for the calibration plate and landmarks. Separate neural networks are then trained to extract the calibration plate and landmarks. A pre-trained YOLOv5 network is used to identify the calibration plate portion and crop it from the original image. The maximum convex hull of the calibration plate is then obtained by sorting the initially extracted landmarks.

[0120] In an exemplary embodiment, the calibration plate is trained using YOLOv5 and VGG16 networks, and the network parameters are consistent with the default parameters. YOLOv5 is used to perform calibration plate extraction, and VGG16 is used to filter the extracted results to further improve the accuracy.

[0121] In an exemplary embodiment, a method combining YOLOv5 and VGG16 networks can be used to extract the annotation points in the image to be tested. First, the calibration plate is annotated with data, and then the anchor point correction is performed on YOLOv5. During the training process, the small target data of the marker point is weighted fused so that the trained network gives priority to smaller marker point features during the target detection process. For example, in the process of training the calibration marker points, the YOLOv5 training method that tends to detect small objects is adopted, and the anchor points and detection boxes in the YOLOv5 model are adjusted to 5%-10% of the size of the marker plate. In this process, YOLOv5 tends to efficiently extract marker points on the cropped calibration plate.

[0122] In addition, an embodiment of the present invention also provides a parameter calibration device for a binocular fisheye camera.

[0123] Reference Figure 6 , Figure 6 Schematic diagram of the functional modules of the first embodiment of the parameter calibration device for a binocular fisheye camera of the present invention.

[0124] In this embodiment, the parameter calibration device of the binocular fisheye camera includes:

[0125] The calibration plate extraction module 10 is used to obtain at least two test images captured by the binocular camera, extract the calibration plate and landmarks in the test images based on the pre-trained network and deep learning method, and obtain the calibration plate image and target landmarks;

[0126] A marker point sorting module 20 is configured to determine standard parameters of the calibration plate image based on a sorting result of the target marker points;

[0127] The calibration plate correction and projection module 30 is used to correct the calibration plate image based on the dedistortion formula to obtain a corrected calibration plate, and project the corrected calibration plate onto the image to be measured based on a preset projection algorithm to obtain a projected calibration plate;

[0128] An internal parameter calibration module 40 is used to calibrate the image to be measured based on the projection calibration plate and the standard parameters to obtain the target internal parameters of the binocular camera, and iteratively optimize the target internal parameters based on an iterative optimization method to obtain calibration internal parameters;

[0129] The extrinsic parameter calibration module 50 is used to obtain an extrinsic parameter calibration field based on the calibration plate extracted from at least two test images captured by the binocular camera, and calculate the calibration extrinsic parameters of the binocular camera based on the extrinsic parameter calibration field and the calibration intrinsic parameters.

[0130] Furthermore, the parameter calibration device for the binocular fisheye camera further includes a dedistortion formula generation module, and the dedistortion formula generation module specifically includes:

[0131] A calibration plate projection calculation unit is used to calibrate the calibration plate based on a fisheye camera model to obtain an initial intrinsic parameter matrix and initial distortion parameters;

[0132] The dedistortion formula generating unit is configured to generate the dedistortion formula based on the initial intrinsic parameter matrix and the initial distortion parameter.

[0133] Furthermore, the calibration plate correction and projection module 30 specifically includes:

[0134] a calibration plate convex hull obtaining unit, configured to obtain the calibration plate convex hull based on the calibration plate image;

[0135] Dedistorting the convex hull to obtain the monocular, and performing dedistortion calculation on the convex hull of the calibration plate based on the dedistortion formula to obtain the dedistorted convex hull;

[0136] The calibration plate obtaining unit is configured to map the dedistorted convex hull based on a homography matrix to obtain the calibration plate.

[0137] Furthermore, the calibration plate correction and projection module 30 specifically includes:

[0138] A calibration plate projection unit is configured to perform projection calculation on the calibration plate based on the fisheye camera model to obtain a current intrinsic parameter matrix and current distortion parameters;

[0139] a calibration plate correction unit, configured to generate a current dedistortion formula based on the current intrinsic parameter matrix and the current distortion parameter, and perform dedistortion correction on the calibration plate based on the current dedistortion formula to obtain a current calibration plate;

[0140] an iterative parameter generation unit, configured to perform projection calculation on the current calibration plate based on the fisheye camera model, generate an iterative intrinsic parameter matrix and iterative distortion parameters, and perform intrinsic parameter calibration of the fisheye camera based on the iterative intrinsic parameter matrix, the iterative distortion parameters, and an iterative optimization method to obtain current calibration internal parameters;

[0141] The calibration plate correction iteration unit is used to generate an iterative dedistortion formula based on the iterative internal parameter matrix and the iterative distortion parameter, perform dedistortion correction on the current calibration plate, generate an iterative calibration plate, and use it as the current calibration plate until the calibration accuracy of the current calibration plate reaches a preset accuracy, use the current calibration internal parameter as the calibration internal parameter, and use the current calibration plate as the correction calibration plate.

[0142] Furthermore, the external parameter calibration module 50 specifically includes:

[0143] A relative pose calculation unit, configured to calculate an estimated relative pose of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameters;

[0144] The calibration extrinsic parameter obtaining unit is used to iteratively optimize the relative pose estimation value based on a preset iterative optimization algorithm to obtain the target pose as the calibration extrinsic parameter of the binocular camera.

[0145] Furthermore, the calibration extrinsic parameter obtaining unit specifically includes:

[0146] An observation coordinate obtaining subunit, configured to transform the coordinate reference point in the external parameter calibration field into the image to be measured based on a coordinate transformation algorithm, and obtain the observation coordinates of the coordinate reference point;

[0147] a current posture change determination subunit, configured to calculate the calculated coordinates of the coordinate reference point based on the calibration internal parameters, and determine the current posture change based on the observed coordinates and the calculated coordinates;

[0148] A current posture obtaining subunit is used to adjust the calibration extrinsic parameter based on the current posture change to obtain the current posture as the calibration extrinsic parameter;

[0149] The calibration extrinsic parameter adjustment subunit is used to transform the coordinate reference point into the image to be measured based on the coordinate transformation algorithm to obtain the observed coordinates, and calculate the current posture change based on the observed coordinates and the calculated coordinates until the current posture change reaches a preset range. Based on the current posture change, the calibration extrinsic parameter is adjusted to obtain the current posture as the calibration extrinsic parameter.

[0150] Furthermore, the calibration plate extraction module 10 specifically includes:

[0151] A data annotation unit, configured to annotate the image to be tested based on a preset annotation method to obtain calibration plate data and landmark point data;

[0152] A calibration plate extraction unit is used to train the calibration plate data and the marker point data based on the pre-trained network and the deep learning method, and extract the calibration plate image and the target marker points in the calibration plate image.

[0153] Among them, each module in the above-mentioned binocular fisheye camera parameter calibration device corresponds to each step in the above-mentioned binocular fisheye camera parameter calibration method embodiment, and their functions and implementation processes are not repeated here one by one.

[0154] In addition, an embodiment of the present invention further provides a computer-readable storage medium.

[0155] The computer-readable storage medium of the present invention stores a binocular fisheye camera parameter calibration program, wherein when the binocular fisheye camera parameter calibration program is executed by a processor, the steps of the binocular fisheye camera parameter calibration method as described above are implemented.

[0156] The method implemented when the binocular fisheye camera parameter calibration program is executed can refer to the various embodiments of the binocular fisheye camera parameter calibration method of the present invention, and will not be repeated here.

[0157] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0158] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0159] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0160] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0161] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A parameter calibration method for a binocular fisheye camera, characterized in that: include: Obtain at least two images to be tested captured by a binocular camera, extract the calibration plate and landmarks in the images to be tested based on a pre-trained network and a deep learning method, and obtain a calibration plate image and target landmarks; Determining standard parameters of the calibration plate image based on a result of sorting the target landmark points; Correcting the calibration plate image based on a dedistortion formula to obtain a corrected calibration plate, and projecting the corrected calibration plate onto the image to be measured based on a preset projection algorithm to obtain a projected calibration plate; Based on the projection calibration plate and the standard parameters, the image to be measured is calibrated to obtain the target internal parameters of the binocular camera, and the target internal parameters are iteratively optimized based on an iterative optimization method to obtain calibration internal parameters; Obtaining an extrinsic calibration field based on the calibration plate extracted from at least two test images captured by the binocular camera, and calculating the calibration extrinsic parameters of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameters; The step of calculating the calibration extrinsic parameters of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameters includes: Calculating a relative pose estimate of the binocular camera based on the extrinsic calibration field and the calibration intrinsic parameter; Iteratively optimizing the relative pose estimation value based on a preset iterative optimization algorithm to obtain a target pose as the calibration extrinsic parameter of the binocular camera; The iterative optimization of the relative pose estimation value based on a preset iterative optimization algorithm to obtain the target pose as the calibration extrinsic parameter of the binocular camera includes: Based on a coordinate transformation algorithm, the coordinate reference point in the external parameter calibration field is transformed into the image to be measured to obtain the observed coordinates of the coordinate reference point; Calculating the calculated coordinates of the coordinate reference point based on the calibration internal parameters, and determining the current posture change based on the observed coordinates and the calculated coordinates; Adjusting the calibration extrinsic parameter based on the current posture change to obtain the current posture as the calibration extrinsic parameter; Based on the coordinate transformation algorithm, the coordinate reference point is transformed into the image to be measured to obtain the observed coordinates, and based on the observed coordinates and the calculated coordinates, the current posture change is calculated until the current posture change reaches a preset range. Based on the current posture change, the calibration external parameter is adjusted to obtain the current posture as the calibration external parameter.

2. The parameter calibration method of a binocular fisheye camera according to claim 1, characterized in that: Before correcting the calibration plate image based on the dedistortion formula to obtain the corrected calibration plate, the method further includes: Based on the fisheye camera model, the calibration plate is calibrated to obtain an initial intrinsic parameter matrix and initial distortion parameters; The dedistortion formula is generated based on the initial internal parameter matrix and the initial distortion parameter.

3. The parameter calibration method of a binocular fisheye camera according to claim 2, characterized in that: The step of correcting the calibration plate image based on the dedistortion formula to obtain the corrected calibration plate includes: Based on the calibration plate image, obtaining a calibration plate convex hull; Based on the dedistortion formula, dedistortion calculation is performed on the convex hull of the calibration plate to obtain a dedistorted convex hull; Based on the homography matrix, the dedistorted convex hull is mapped to obtain the correction calibration plate.

4. The method for calibrating parameters of a binocular fisheye camera according to any one of claims 1 to 3, characterized in that: The method of extracting the calibration plate and the landmark points in the image to be tested based on the pre-trained network and the deep learning method to obtain the calibration plate image and the target landmark points includes: Based on a preset annotation method, data annotation is performed on the image to be tested to obtain calibration plate data and landmark point data; Based on the pre-trained network and the deep learning method, the calibration plate data and the marker point data are trained to extract the calibration plate image and the target marker points in the calibration plate image.

5. A parameter calibration device for a binocular fisheye camera, characterized in that: The parameter calibration device of the binocular fisheye camera includes: A calibration plate extraction module is used to obtain at least two test images captured by the binocular camera, and extract the calibration plate and landmarks in the test images based on a pre-trained network and a deep learning method to obtain a calibration plate image and target landmarks; A marker point sorting module, configured to determine standard parameters of the calibration plate image based on a sorting result of the target marker points; A calibration plate correction and projection module is used to correct the calibration plate image based on a dedistortion formula to obtain a corrected calibration plate, and project the corrected calibration plate onto the image to be measured based on a preset projection algorithm to obtain a projected calibration plate; An internal parameter calibration module is used to calibrate the image to be measured based on the projection calibration plate and the standard parameters to obtain the target internal parameters of the binocular camera, and iteratively optimize the target internal parameters based on an iterative optimization method to obtain calibration internal parameters; An extrinsic parameter calibration module is configured to obtain an extrinsic parameter calibration field based on the calibration plate extracted from at least two test images captured by the binocular camera, and calculate the calibration extrinsic parameters of the binocular camera based on the extrinsic parameter calibration field and the calibration intrinsic parameters; The extrinsic parameter calibration module is specifically used to calculate the relative pose estimation value of the binocular camera based on the extrinsic parameter calibration field and the calibration intrinsic parameter; based on a preset iterative optimization algorithm, the relative pose estimation value is iteratively optimized to obtain the target pose as the calibration extrinsic parameter of the binocular camera; The iterative optimization of the relative pose estimation value based on a preset iterative optimization algorithm to obtain the target pose as the calibration extrinsic parameter of the binocular camera includes: Based on a coordinate transformation algorithm, the coordinate reference point in the external parameter calibration field is transformed into the image to be measured to obtain the observed coordinates of the coordinate reference point; Calculating the calculated coordinates of the coordinate reference point based on the calibration internal parameters, and determining the current posture change based on the observed coordinates and the calculated coordinates; Adjusting the calibration extrinsic parameter based on the current posture change to obtain the current posture as the calibration extrinsic parameter; Based on the coordinate transformation algorithm, the coordinate reference point is transformed into the image to be measured to obtain the observed coordinates, and based on the observed coordinates and the calculated coordinates, the current posture change is calculated until the current posture change reaches a preset range. Based on the current posture change, the calibration external parameter is adjusted to obtain the current posture as the calibration external parameter.

6. A parameter calibration device for a binocular fisheye camera, characterized in that: The binocular fisheye camera parameter calibration device includes a processor, a memory, and a binocular fisheye camera parameter calibration program stored in the memory and executable by the processor, wherein when the binocular fisheye camera parameter calibration program is executed by the processor, the steps of the binocular fisheye camera parameter calibration method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a binocular fisheye camera parameter calibration program, wherein when the binocular fisheye camera parameter calibration program is executed by a processor, the steps of the binocular fisheye camera parameter calibration method according to any one of claims 1 to 4 are implemented.

Images