A camera calibration method, device, apparatus and storage medium

Abstract

This disclosure provides a camera calibration method, apparatus, device, and storage medium to improve calibration success rate and result stability. The camera calibration method includes: acquiring the image coordinates and spatial coordinates of each corner point in multiple frames of calibration images captured by the camera to be calibrated; employing various pose estimation methods, estimating the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration images and the initial focal length of the camera to be calibrated based on the image coordinates and spatial coordinates, obtaining multiple sets of initial estimation results; combining the initial focal lengths from each set of initial estimation results with multiple preset model parameter values, using the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length, determining a set of candidate calibration parameters for each combination result; substituting each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point; and selecting the target calibration parameter as the calibration result based on the reprojection error corresponding to each set of candidate calibration parameters.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to camera calibration methods, apparatus, devices, and storage media. Background Technology

[0002] Wide field-of-view fisheye cameras are widely used in robotics, autonomous driving, security monitoring, and other fields. The projection relationship of fisheye cameras is highly nonlinear, and a unified camera model (UCM) can generally be used to model the projection relationship of fisheye cameras.

[0003] Camera calibration typically involves capturing multiple frames of a calibration target image with known 3D coordinates using the camera to be calibrated. Based on the projection relationship between the detected corner pixel positions and their corresponding spatial points, the intrinsic parameters of the camera and the extrinsic parameters (i.e., pose, including rotation and translation) for each frame are calculated. Once accurate calibration parameters are obtained, distortion correction and rectification can be performed on the images acquired by the fisheye camera, resulting in distortion-corrected images for subsequent vision tasks.

[0004] In solving for the aforementioned intrinsic and extrinsic parameters, it is necessary to first obtain a set of initial values ​​including both intrinsic and extrinsic parameters, and then substitute them into the optimizer for iterative refinement. Since the calibration process is sensitive to initial values, it is prone to getting trapped in local minima during optimization, leading to calibration failure. Furthermore, some initial intrinsic parameter values ​​deviate from the correct parameter range, further exacerbating the convergence difficulty. Summary of the Invention

[0005] This disclosure provides a camera calibration method, apparatus, device, and storage medium to improve calibration success rate and calibration result stability.

[0006] In a first aspect, a camera calibration method is provided, comprising: acquiring the image coordinates and corresponding spatial coordinates of each corner point in multiple frames of calibration images captured by the camera to be calibrated; employing multiple different pose estimation methods, respectively estimating the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration images and the initial focal length of the camera to be calibrated based on the image coordinates and the spatial coordinates, to obtain multiple sets of initial estimation results; combining the initial focal length in each set of initial estimation results with multiple preset model parameter values, and constraining that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length, determining a set of candidate calibration parameters for each combination result; substituting each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point, and selecting the target calibration parameter as the calibration result from each set of candidate calibration parameters based on the reprojection error corresponding to each set of candidate calibration parameters.

[0007] In one embodiment, the step of selecting a target calibration parameter as the calibration result from each set of candidate calibration parameters based on the reprojection error corresponding to each set of candidate calibration parameters includes: determining the reprojection error statistics corresponding to each set of candidate calibration parameters based on the reprojection error of each corner point corresponding to each set of candidate calibration parameters; selecting a preset number of sets of candidate calibration parameters based on the reprojection error statistics corresponding to each set of candidate calibration parameters, and independently performing nonlinear optimization for each set of selected candidate calibration parameters; and selecting a target calibration parameter as the calibration result from the preset number of sets of candidate calibration parameters based on the reprojection error statistics of each corner point corresponding to each set of candidate calibration parameters after optimization.

[0008] In one implementation, the step of independently performing nonlinear optimization for each selected set of candidate calibration parameters includes: in a first stage, using a squared loss function as the optimization objective, jointly optimizing the intrinsic parameters of the candidate calibration parameters and the extrinsic parameters of each frame of calibration image until the parameter change is less than a preset threshold; in a second stage, switching the optimization objective to a Cauchy bar kernel, and in each iteration, based on the statistical distribution of the reprojection errors of all current corner points, eliminating corner points whose single-point reprojection errors exceed a preset multiple of the standard deviation, and continuing to jointly optimize the intrinsic and extrinsic parameters of the candidate calibration parameters; and outputting the intrinsic and extrinsic parameters of the candidate calibration parameters after optimization in the first and second stages.

[0009] In one embodiment, the step of selecting a preset number of candidate calibration parameters based on the reprojection error statistics corresponding to each group of candidate calibration parameters, and independently performing nonlinear optimization for each selected group of candidate calibration parameters, includes: sorting the candidate calibration parameters corresponding to the multiple initial estimation results in ascending order of the reprojection error statistics corresponding to each group of candidate calibration parameters; based on the sorting results, starting from the group with the smallest reprojection error statistics, sequentially selecting candidate calibration parameters in ascending order and performing the nonlinear optimization until the optimization of the preset number of candidate calibration parameters is completed; wherein, when the reprojection error statistics of any group of candidate calibration parameters after optimization is less than or equal to a preset threshold, the optimization of subsequent groups of candidate calibration parameters is stopped, and the optimized result of that group of candidate calibration parameters is output as the target calibration parameter.

[0010] In one embodiment, the various pose estimation methods include a distortion-aware estimation method and a pinhole estimation method; the distortion-aware estimation method includes: using a fisheye distortion model and estimating the extrinsic parameters and the initial focal length using all corner points; the pinhole estimation method includes a first sub-method and a second sub-method; the first sub-method includes: using only corner points located in the central region of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model; the second sub-method includes: using all corner points of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model.

[0011] In one embodiment, determining a set of candidate calibration parameters for each combination result includes: for each combination result, performing a scaling transformation on the initial focal length based on the model parameter values ​​in the combination result to obtain in-focal-length parameter values; the scaling transformation is used to make the equivalent focal length of the imaging model in the paraxial region equal to the initial focal length; and using the in-focal-length parameter values, the model parameter values, preset initial principal point values, preset initial distortion coefficient values, and the extrinsic parameters in the initial estimation result as the candidate calibration parameters.

[0012] In one embodiment, the camera to be calibrated is a fisheye camera; the imaging model of the camera to be calibrated is a unified camera model; the unified camera model is superimposed with radial distortion to describe the projection relationship of the fisheye camera; and the model parameters are the geometric control parameters of the unified camera model.

[0013] In one embodiment, the plurality of preset model parameter values ​​include a plurality of preset distortion degree parameter values; the plurality of preset distortion degree parameter values ​​are arranged in ascending or descending order and cover the effective value range of the imaging model to characterize different degrees of fisheye distortion.

[0014] In one embodiment, the method further includes: applying the method to a multi-eye fisheye system containing at least two fisheye cameras, executing the method on each fisheye camera in the multi-eye fisheye system to obtain the intrinsic and extrinsic parameters of each fisheye camera; utilizing the inter-eye co-view constraint formed by the same calibration target appearing simultaneously in the fields of view of at least two fisheye cameras at the same time, and using the intrinsic and extrinsic parameters of each fisheye camera as initial values, performing joint global optimization on the intrinsic parameters of all fisheye cameras and the relative extrinsic parameters between the cameras.

[0015] In one embodiment, the method further includes: during the acquisition of the multi-frame calibration images, evaluating the spatial distribution of the extrinsic parameters of the calibration target in each frame of calibration images based on the image coordinates of the corner points and their corresponding spatial coordinates in the acquired calibration images, and statistically analyzing the degree of constraint of the acquired calibration images in each degree of freedom; when the degree of constraint of any degree of freedom is lower than a preset threshold, generating guidance information, which is used to prompt the operator to adjust the orientation of the calibration target and acquire additional calibration image frames; after acquiring the additional calibration image frames acquired based on the guidance information, continuing to execute the step of acquiring the image coordinates of each corner point and their corresponding spatial coordinates in the multi-frame calibration images captured by the camera to be calibrated.

[0016] In one embodiment, the method further includes: after obtaining the target calibration parameters, calculating the reprojection error for all corner points used for calibration and statistically analyzing the distribution of the reprojection error to form an error distribution statistical result; calculating the parameter covariance matrix based on the loss function value at nonlinear optimization convergence, and obtaining the confidence interval of each parameter in the calibration result from the parameter covariance matrix; weighting the error distribution statistical result with the confidence interval of each parameter to obtain a calibration confidence score; and generating diagnostic prompt information when the calibration confidence score is lower than a preset score threshold.

[0017] Secondly, a camera calibration device is provided, comprising:

[0018] The acquisition module is used to acquire the image coordinates of each corner point and its corresponding spatial coordinates in multiple frames of calibration images captured by the camera to be calibrated. The initial estimation module is used to estimate the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration image and the initial focal length of the camera to be calibrated, respectively, using a variety of different pose estimation methods, based on the image coordinates and the spatial coordinates, to obtain multiple sets of initial estimation results; The candidate calibration module is used to combine the initial focal length in each set of initial estimation results with multiple preset model parameter values, and determine a set of candidate calibration parameters for each combination result, with the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length. The target calibration module is used to substitute each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point, and select the target calibration parameter as the calibration result from each set of candidate calibration parameters based on the reprojection error corresponding to each set of candidate calibration parameters.

[0019] Thirdly, an apparatus is provided, including a processor and a memory, wherein computer instructions are stored in the memory, and the processor is configured to execute the computer instructions to perform the steps of the camera calibration method described in any of the above embodiments.

[0020] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the camera calibration method as described in any of the above embodiments.

[0021] The embodiments of this disclosure employ various pose estimation methods to estimate extrinsic parameters and initial focal length, resulting in multiple sets of initial estimation results. Different pose estimation methods have different bias directions, which can cover the parameter space from multiple angles, effectively increasing the probability that at least one set of estimation results falls into the correct parameter region, thereby significantly reducing the risk of calibration failure due to initial value deviation.

[0022] Furthermore, in this embodiment, the initial focal length in each set of initial estimation results is combined with multiple preset model parameter values, and each set of candidate calibration parameters is determined by the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length. This ensures that the initial values ​​of the constructed complete calibration parameters are consistent with the geometric scale implied by the pose estimator in terms of projection scale, thereby making the optimization starting point closer to the real parameter region, accelerating the convergence speed and reducing the number of iterations.

[0023] Finally, the reprojection error of each group of candidate calibration parameters is calculated under the projection relationship of the imaging model, and the target calibration parameter is selected accordingly. By comparing and screening candidate calibration parameters from different sources, a more stable and reliable calibration result is finally output.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0025] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0027] Figure 1 A flowchart of a camera calibration method provided as an exemplary embodiment of this disclosure; Figure 2A flowchart of a camera calibration method provided as another exemplary embodiment of this disclosure; Figure 3 A schematic diagram of a camera calibration device 300 provided for an exemplary embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a camera calibration device 400 provided for an exemplary embodiment of the present disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0029] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0030] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0031] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0032] The calibration of a fisheye camera can be described by superimposing a unified camera model with radial distortion to represent its projection relationship. For example, this model can include seven intrinsic parameters: two focal length components. (Horizontal equivalent focal length) (Longitudinal equivalent focal length), principal point (x-coordinate of the principal point) (Principal point ordinate), unified model parameters and second-order radial distortion coefficient and fourth-order radial distortion coefficient During calibration, multiple frames of calibration target images with known 3D coordinates are captured using the camera to be calibrated. Based on these images, the seven intrinsic parameters and the corresponding extrinsic parameters for each frame are solved. Because the model has many parameters and the projection relationship is nonlinear, the optimization process is highly sensitive to initial values.

[0033] When obtaining initial values, an initial focal length and corresponding extrinsic parameters can be obtained through pose estimation. This focal length can then be directly assigned to the focal length component of the model, while manually setting... The initial values ​​are used to construct the complete set of initial intrinsic parameters before being fed into the optimizer. However, in this approach, if the estimation method becomes biased, the optimization is prone to getting trapped in local minima, leading to calibration failure. Furthermore, direct assignment ignores the focal length component and its relationship to the model. The coupling relationship causes the constructed initial intrinsic parameters to be inconsistent with the scale on which the pose estimation is based in terms of projection scale, resulting in the optimization starting point deviating from the correct parameter region and making convergence difficult.

[0034] Based on this, this disclosure provides a camera calibration method. The core idea is as follows: First, multiple initial focal lengths and extrinsic parameters are obtained using various pose estimation methods. Then, each initial focal length is combined with multiple preset model parameter values. The equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length as a constraint to determine each set of candidate calibration parameters. For each candidate calibration parameter, the reprojection error is calculated uniformly under the imaging model, and the target calibration parameter is selected as the final calibration result based on the reprojection error. Using the calibration method of this disclosure can reduce the sensitivity of calibration to initial values ​​and improve the calibration success rate and result stability.

[0035] like Figure 1 The diagram shows a flowchart of a camera calibration method provided in an exemplary embodiment of this disclosure. This method can be applied to mobile platforms such as robots and vehicles equipped with cameras, or to calibration station equipment used to calibrate such cameras. The camera is, for example, a fisheye camera. The imaging model of the camera to be calibrated can be a unified camera model, which can be used to describe the strongly nonlinear projection relationship of the fisheye camera. The calibration method includes: S101: Obtain the image coordinates of each corner point and its corresponding spatial coordinates in the multi-frame calibration images captured by the camera to be calibrated.

[0036] For example, in an environment containing multiple non-coplanar calibration targets, multiple frames of calibration images are acquired using a fisheye camera to be calibrated. The calibration targets can be checkerboard calibration boards, dot array calibration boards, or other artificial calibration objects with known three-dimensional coordinate distributions. The corner points on the calibration targets are detected (e.g., using a checkerboard corner detection algorithm or a dot center detection algorithm) to obtain the two-dimensional pixel observation coordinates (image coordinates) of the corner points in each frame of the calibration image and their corresponding three-dimensional spatial point coordinates (spatial coordinates). That is, the three-dimensional position of a certain spatial point in the coordinate system of the calibration target and its observed pixel position in that frame of the image are known.

[0037] In this embodiment of the disclosure, the imaging model of the camera to be calibrated can be a Unified Camera Model (UCM), which is suitable for describing the projection relationships of large field-of-view imaging devices such as fisheye cameras. Within the UCM framework, three-dimensional spatial points in the camera coordinate system are first represented by a model with parameters defined by the unified model. The controlled normalized projection is mapped onto the normalized image plane, and then mapped to pixel coordinates via the focal length component and principal point. To further correct residual distortion, a radial distortion correction term can be superimposed on this. The imaging model can contain multiple distortion coefficients (such as second-order and fourth-order distortion coefficients), which can more accurately describe the nonlinear projection characteristics of a fisheye camera at a large field of view.

[0038] It is understood that the radial distortion model described above is merely an example. In other embodiments, radial distortion can be extended to higher orders (such as sixth or eighth order) or a tangential distortion term can be added. The imaging model can also be generalized from UCM to variant models of UCM, such as the Extended Unified Model (EUCM) or the Double Sphere model, without departing from the technical concept of this disclosure. In the embodiments of this disclosure, UCM and its variant models are collectively referred to as the Unified Camera Model.

[0039] S102: Using various pose estimation methods, based on the image coordinates and the spatial coordinates, estimate the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration image and the initial focal length of the camera to be calibrated, and obtain multiple sets of initial estimation results.

[0040] For example, the correspondence between image coordinates and spatial coordinates obtained in step S101 is processed using at least two pose estimation methods with different structures. Each pose estimation method estimates two aspects of information based on the same observation data (image coordinates of each corner point and their corresponding spatial coordinates): first, the extrinsic parameters of the camera relative to the calibration object in each frame of the calibration image (i.e., rotation matrix and translation vector, i.e., pose); and second, the initial focal length of the camera. Due to the different principles and data usage methods of the different estimation methods, the estimation results obtained by each method differ numerically, thus forming multiple sets of initial estimation results.

[0041] For example, one pose estimation method can employ a calibration approach incorporating a fisheye distortion model, jointly estimating focal length and extrinsic parameters using all corner points (including strongly distorted areas at the edges of the image). Another pose estimation method can use a pinhole model, calculating pose and focal length only using corner points in the central region of the image (where distortion is minimal). A global pinhole model can also be used, calculating pose and focal length using all corner points. These methods operate independently, each outputting a set of initial estimation results—a set of estimated values ​​including the initial focal length and extrinsic parameters for each frame.

[0042] S103: Combine the initial focal lengths in each set of initial estimation results with multiple preset model parameter values, and determine a set of candidate calibration parameters for each combination result, with the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length.

[0043] For example, for each set of initial estimation results obtained in step S102, its initial focal length is combined with multiple preset model parameter values, and each combination result includes an initial focal length and a preset model parameter. For each combination, a complete set of candidate calibration parameters is determined according to specific constraints.

[0044] For example, the preset model parameter values ​​can be the uniform model parameters of the UCM model. The values ​​of (also known as geometric control parameters). It is a key parameter for controlling the degree of distortion in a fisheye lens, and its value reflects the size of the lens's field of view. A larger value indicates stronger distortion and a wider field of view. Multiple preset values ​​are available. The values ​​can be a set of seed values ​​arranged in ascending or descending order, covering the fisheye intensity from weak to strong, such as {0, 0.2, 0.4, 0.6, 0.8, 1.0}.

[0045] For the initial focal length in each set of initial estimation results This is compared with each preset model parameter value. By combining, multiple results can be obtained. Combinations. For each combination, determine the complete candidate calibration parameters for the UCM model. During this determination process, the equivalent focal length of the initial UCM intrinsic parameters constructed by the constraints in the paraxial region is compared with the initial focal length upon which the pose estimation in the initial estimation result is based. Maintain consistency in scale.

[0046] The paraxial region refers to the minimal field of view near the camera's optical axis (i.e., a small field of view near the image center). Within this region, the projection relationship of the unified camera model can be approximated as a pinhole model. That is, for three-dimensional points in the corresponding camera coordinate system, z >> x, y, which is the region where the imaging light rays are close to the optical axis. At this point, the influence of fisheye distortion approaches zero, and the projection behavior of the UCM model can be approximated as an ideal pinhole model. As an example, the minimal field of view region is centered on the image principal point and covers approximately [number missing] pixels of the total image. The central region, or the field of view angle is smaller than The imaging area. Within this area, the projection characteristics of an actual fisheye camera tend to approximate a pinhole model. It should be noted that the above range is only a preferred example of this disclosure. Those skilled in the art can select an appropriate range based on the camera model, lens parameters, and calibration accuracy requirements, all of which fall within the paraxial region defined in this disclosure.

[0047] Based on first-order approximation analysis, the equivalent focal length of UCM in the paraxial region is... ;in The focal length component of the UCM model (for simplicity, here) ), To standardize model parameters, this formula shows that the equivalent focal length of UCM in the paraxial region is not directly equal to the focal length component. , but by and A joint decision.

[0048] Therefore, if a pose estimation method uses the pinhole model assumption to estimate extrinsic parameters and obtains an initial focal length f, then this initial focal length f represents the equivalent focal length scale under the paraxial pinhole approximation, and the extrinsic parameters are also estimated under this scale. If f is directly assigned to the focal length component of the UCM... Therefore, the paraxial equivalent focal length of UCM is actually... This is inconsistent with the scale f upon which pose estimation is based; there is a dependency between the two. Systematic bias.

[0049] Therefore, in determining candidate calibration parameters for each combination result in this embodiment of the disclosure, the constraint is that the paraxial equivalent focal length is equal to the initial focal length, that is: Therefore, the focal length intrinsic parameter is determined. This method of setting the value ensures that the equivalent focal length of the UCM intrinsic parameter in the paraxial region is exactly equal to the initial focal length f given by the pose estimator, thereby eliminating the deviation between the projection scale implied by the initial intrinsic parameter value and the geometric scale represented by the extrinsic parameter, allowing the intrinsic and extrinsic parameters to enter subsequent optimization under the same geometric scale.

[0050] Based on this, the focal length intrinsic parameter determined in the above manner will be... and (During the initial value construction phase, in order to simplify and maintain isotropy, we can let) ), and the model parameter values ​​in the current combination The preset initial value of the principal point (usually taken as the geometric center of the image), the preset initial value of the radial distortion coefficient (usually taken as zero), and the corresponding extrinsic parameter estimation results in step S102 together constitute a complete set of candidate calibration parameters.

[0051] Assuming that step S102 yields P initial estimation results (P≥2), with each group assigned Q preset model parameter values, this step generates a total of P×Q candidate calibration parameters, forming a candidate calibration parameter set.

[0052] S104: Substitute each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point, and select the target calibration parameter from each set of candidate calibration parameters as the calibration result based on the reprojection error corresponding to each set of candidate calibration parameters.

[0053] In practice, all candidate calibration parameters generated in step S103 are evaluated under the UCM (including radial distortion) projection relationship. For example, for each set of candidate calibration parameters, the intrinsic and extrinsic parameters are substituted into the UCM projection model to calculate the predicted pixel coordinates for all corner points obtained in step S101. Then, the predicted pixel coordinates are compared with the actually detected pixel observation coordinates to calculate the reprojection error for each corner point. The reprojection error reflects the degree of model fit to the observed data under that set of calibration parameters. The smaller the error, the better the consistency between the candidate calibration parameters and the observed data.

[0054] Since all candidate calibration parameters are calculated for reprojection error under the same UCM projection relationship, candidates from different sources are uniformly mapped to the same metric, allowing for direct comparison and ranking. For example, based on the reprojection error statistics corresponding to each group of candidate calibration parameters (e.g., the root mean square of the reprojection error of all corner points of each candidate, or the mean of the reprojection error of all corner points of each candidate), one or more groups of candidates with the smallest error can be selected, and their calibration parameters can be output as target calibration parameters to complete the calibration. This target calibration parameter is the final calibration result, which can be output for subsequent distortion correction and rectification of images acquired by fisheye cameras, or for tasks such as multi-sensor fusion perception in robots.

[0055] In some embodiments, in order to ensure calibration robustness while taking into account computational efficiency and further avoid the risk of possible local optima, in some embodiments of this disclosure, the process of selecting the target calibration parameter from each group of candidate calibration parameters in step S104 is not simply based on the initial reprojection error sorting and directly selecting the best, but adopts a multi-level selection mechanism of initial screening, optimization and selection of the best.

[0056] Based on this, such as Figure 2 As shown, S104 selects the target calibration parameter as the calibration result from each group of candidate calibration parameters based on the reprojection error corresponding to each group of candidate calibration parameters, which may include: S104a: Based on the reprojection error of each corner point corresponding to each group of candidate calibration parameters, determine the statistical value of the reprojection error corresponding to each group of candidate calibration parameters.

[0057] S104b: Based on the reprojection error statistics corresponding to each group of candidate calibration parameters, select a preset number of candidate calibration parameters, and perform nonlinear optimization independently for each selected group of candidate calibration parameters.

[0058] S104c: Based on the reprojection error statistics of each corner point corresponding to each group of candidate calibration parameters after optimization, select the target calibration parameter as the calibration result from the preset number of candidate calibration parameters.

[0059] For example, in the process of selecting the target calibration parameter from each group of candidate calibration parameters as the calibration result, this embodiment does not directly use the group of candidates with the smallest reprojection error as the final calibration result. Instead, it uses a three-stage process of initial screening, optimization, and selection to ensure the reliability and accuracy of the final result. For instance, in step S104, each group of candidate calibration parameters is first substituted into the UCM imaging model to calculate the reprojection error of each group of candidates at all corner points, and the corresponding statistical value of the reprojection error for each group of candidate calibration parameters is determined accordingly. This statistical value can be the root mean square error (RMSE) of the reprojection error of each candidate at all corner points, or it can be the mean of the reprojection error or other statistics that can reflect the overall error level. Through this statistical value, all candidate calibration parameters can be uniformly quantitatively evaluated and ranked.

[0060] Subsequently, based on the reprojection error statistics corresponding to each group of candidate calibration parameters, a predetermined number (e.g., the top K, where K is a configurable positive integer, such as K=5 or K=10) of candidate calibration parameters with the smallest reprojection error are selected. For each selected group of candidate calibration parameters, a nonlinear optimization is performed independently. The reason for optimizing multiple groups of candidates separately, rather than just optimizing the group with the highest score, is that although the candidate with the highest reprojection error performs best under the current metric, the nonlinear optimization problem has multiple local minima, and the top-ranked candidate may not remain optimal after optimization. Optimizing multiple groups of candidates separately before selecting the best one increases the probability of finally selecting the globally optimal solution. The optimization of each group of candidates is independent and can be performed in parallel to improve efficiency. The goal of the nonlinear optimization is to jointly refine the seven-dimensional intrinsic parameters and the extrinsic parameters of each frame in the candidate calibration parameters to further reduce the reprojection error.

[0061] Finally, based on the reprojection error statistics of each corner point corresponding to the optimized candidate calibration parameters, the optimal set is selected from this preset number of candidate calibration parameters as the target calibration parameters. This mechanism of initial screening, optimization, and selection ensures that the final calibration result is generated after independent optimization and objective comparison among multiple potential candidates, thus possessing higher reliability and stability.

[0062] In some embodiments, in order to ensure that each group of candidate calibration parameters can converge quickly during the nonlinear optimization process and effectively suppress the interference caused by false corner detections, the above-mentioned independent optimization process can be executed as a progressive process including a warm-up stage and a robust refinement stage in some embodiments.

[0063] Based on this, the independent nonlinear optimization for each selected set of candidate calibration parameters may include: in the first stage, using the squared loss function as the optimization objective, jointly optimizing the intrinsic parameters of the candidate calibration parameters and the extrinsic parameters of each frame of calibration image until the parameter change is less than a preset threshold; in the second stage, switching the optimization objective to the Cauchy bar kernel, and in each iteration, based on the statistical distribution of the reprojection errors of all current corner points, eliminating corner points whose single-point reprojection errors exceed a preset multiple of the standard deviation, and continuing to jointly optimize the intrinsic and extrinsic parameters of the candidate calibration parameters; and outputting the intrinsic and extrinsic parameters of the candidate calibration parameters after optimization in the first and second stages.

[0064] For example, a two-stage optimization strategy is adopted for the nonlinear optimization performed independently for each selected set of candidate calibration parameters, so as to balance stable convergence in the early stage of optimization and robustness in the later stage of optimization.

[0065] In the first stage, the squared loss function (L2 loss) is used as the optimization objective to optimize the seven-dimensional intrinsic parameters among the candidate calibration parameters. , , , , , , The optimization is performed jointly on the extrinsic parameters (rotation matrix and translation vector) of each frame of calibration image.

[0066] The L2 loss function described above weights the errors equally across all corners, exhibiting smooth gradients and stable convergence, making it suitable for quickly bringing parameters from their initial values ​​to a stable region during the initial optimization phase. The formula can be expressed as: ; in, The total number of frames for the calibration image; For the first The total number of corner points involved in optimization in the frame image; For the first The first frame of the image The actual pixel observation coordinates (two-dimensional vector) of each corner point; It is a seven-dimensional intrinsic parameter vector, that is ; For the first The extrinsic parameters corresponding to the frame image, namely the rotation matrix and translation vector; For the first The three-dimensional spatial coordinates of each corner point; To unify the complete projection function of the camera model superimposed radial distortion, the 3D spatial points are transformed from the world coordinate system to the camera coordinate system through extrinsic parameter transformation, and then mapped to the predicted pixel coordinates after unified camera model projection and distortion correction. It is the Euclidean norm, which is the Euclidean distance between the predicted pixel coordinates and the actual pixel coordinates.

[0067] The iteration termination condition for this stage is: iteration stops when the parameter change is less than a preset threshold. This parameter change threshold can be set to, for example, 1 × 10⁻⁶. -8 (That is, the absolute value of the change in each element in the intrinsic and extrinsic parameters in two adjacent iterations is less than 1×10) -8 (), or the relative pose change is less than 1×10 -6 This threshold can be adjusted according to the actual calibration accuracy requirements. The smaller the threshold, the more optimization iterations are needed and the higher the accuracy may be, but the computation time will also increase accordingly.

[0068] In the second stage, the optimization objective is switched from the squared loss function to the Cauchy robust kernel function. The formula can be: ; in, The reprojection error of a single corner point, i.e. ; For the first The first frame in the calibration image The actual pixel observation coordinates of each corner point; It is a seven-dimensional intrinsic parameter vector; , The first The rotation matrix and translation vector corresponding to the frame calibration image; For the first The three-dimensional spatial coordinates of each corner point; To unify the camera model by superimposing a complete projection function for radial distortion; It is the Euclidean norm. The scaling parameter controls the inflection point where the robust kernel transitions from quadratic to logarithmic growth. Its value can be determined based on the estimated value of typical reprojection errors in the image; for example, it can be set to c = 1.0~2.0 pixels. The specific value can be adaptively adjusted according to the resolution of the calibrated image and the corner detection accuracy. The Cauchy kernel is characterized by its behavior approximating squared loss (sensitive to errors) when the single-point error r is small; when r is large, its growth tends to be gradual (insensitive to errors), thus reducing the pulling force of large error points on the optimization direction.

[0069] It is understood that the Cauchy robust kernel described above is merely an example. In other implementations, the robust kernel function can be replaced by other forms of robust kernel functions such as Huber or Tukey, the core of which lies in reducing the impact of large error points on the optimization direction through weight reduction or truncation. Furthermore, in addition to the average reprojection error of all participating points, other implementations may also include the quantiles or maximum values ​​of the error distribution as auxiliary criteria to more comprehensively evaluate the robustness of the calibration results.

[0070] In each iteration of the second phase, corner points whose reprojection errors exceed a preset multiple of the standard deviation are removed based on the statistical distribution of the reprojection errors of all current corner points. For example, the mean of the reprojection errors of all current corner points can be calculated. and standard deviation The error will exceed Points with significant errors are marked as outliers and excluded from optimization in the current iteration. Subsequently, joint optimization of intrinsic and extrinsic parameters continues. It should be noted that the excluded corner points may simply be excluded from optimization in that particular iteration. As the intrinsic and extrinsic parameters gradually improve during optimization, some corner points that initially had large errors may return to the normal error range in subsequent iterations. Therefore, the error distribution can be recalculated and the outlier determination dynamically updated in each iteration.

[0071] The second stage effectively suppresses the interference of outliers on the calibration results introduced by factors such as corner false detection, calibration target occlusion, and unevenness of the calibration target plane by combining soft weight adjustment of Cauchy robust kernel with hard removal of outliers based on statistical distribution. This makes the final calibration parameters more accurately reflect the projection relationship of normal corner points.

[0072] After both stages of optimization are completed, the intrinsic and extrinsic parameters of the candidate calibration parameters after the first and second stages of optimization are output for subsequent stability selection.

[0073] In the two-stage optimization strategy described above, the squared loss warm-up stage enables the parameters to quickly enter the stable region, while the Cauchy robust kernel and outlier removal stage finely adjusts the parameters and resists outlier interference. The combination of the two achieves a smooth transition from rapid convergence to robust refinement.

[0074] In some embodiments, since performing full nonlinear optimization on each group of candidate calibration parameters would result in high computational costs, and the candidates ranked higher have higher optimization potential, in order to improve calibration efficiency, in some embodiments, the steps of selecting a preset number of candidate calibration parameters and performing nonlinear optimization can adopt a progressive strategy of sequential selection, group-by-group optimization, and stopping once the target is reached.

[0075] Based on this, according to the reprojection error statistics corresponding to each group of candidate calibration parameters, a preset number of candidate calibration parameters are selected. Nonlinear optimization is performed independently on each selected group of candidate calibration parameters, including: sorting the candidate calibration parameters corresponding to the multiple initial estimation results in ascending order of the reprojection error statistics corresponding to each group of candidate calibration parameters; based on the sorting results, starting from the group with the smallest reprojection error statistics, selecting candidate calibration parameters in ascending order and performing the nonlinear optimization until the optimization of the preset number of candidate calibration parameters is completed; wherein, when the reprojection error statistics of any group of candidate calibration parameters after optimization is less than or equal to a preset threshold, the optimization of subsequent groups of candidate calibration parameters is stopped, and the optimized result of that group of candidate calibration parameters is output as the target calibration parameter.

[0076] For example, the candidate calibration parameters are first sorted from smallest to largest according to their reprojection error statistics. These statistics can be the root mean square or average of the reprojection errors at all corner points, reflecting the degree of fit between the candidate and the observed data before nonlinear optimization. Then, based on the sorting, starting with the group with the smallest reprojection error statistics, candidate calibration parameters are selected sequentially to perform the two-stage nonlinear optimization described above. Unlike the method of selecting the first K groups of candidates at once and then optimizing them one by one, this method uses a sequential selection and group-by-group optimization strategy: first, the candidate ranked first is optimized, and its result is evaluated; if the result does not meet the preset accuracy requirements, then the candidate ranked second is optimized, and so on, until the preset number of candidate groups have been optimized.

[0077] During this process, if the optimized reprojection error statistics of a certain group of candidate calibration parameters are less than or equal to a preset threshold, the optimization of subsequent candidate groups will be stopped, and the optimized result of that group of candidates will be output as the target calibration parameter. This preset threshold is an acceptable upper bound for reprojection error, for example, it can be set to 0.5 pixels or 1.0 pixels, and the specific value can be adjusted according to the calibration accuracy requirements of the actual application.

[0078] This approach achieves two key benefits. First, by selecting candidates sequentially and optimizing them group by group, the most promising candidates (ranked higher by reprojection error) are prioritized, avoiding unnecessary optimization calculations on a large number of inferior candidates. Second, once the optimization results of a certain group of candidates meet the preset accuracy requirements, subsequent optimizations are terminated early, significantly saving computation time and improving calibration efficiency. Furthermore, the introduction of a preset threshold ensures that the output results have a definite accuracy guarantee, rather than simply relying on candidate ranking. It should be noted that the maximum number of optimizations for the preset number of candidate groups is still constrained by the K value. Even if the first K groups of candidates fail to meet the target, only K groups are optimized, and the best one is selected, ensuring the controllability of the entire process and termination within a limited time.

[0079] In other implementations, if the application scenario has high requirements for calibration accuracy, a preset threshold for stopping once the target is met may not be set. Instead, after all preset number groups of candidates have independently completed two-stage optimization, the best candidate is selected and output based on the average reprojection error of all participating points after optimization, so as to further ensure the optimality of the final calibration result in the global range.

[0080] In some embodiments, the various pose estimation methods mentioned in step S102 may include distortion-aware estimation and pinhole estimation.

[0081] The distortion-aware estimation method includes: employing a fisheye distortion model and estimating the extrinsic parameters and the initial focal length using all corner points. The pinhole estimation method includes a first sub-method and a second sub-method. The first sub-method includes: using only corner points located in the center region of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model; the second sub-method includes: using all corner points of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model.

[0082] For example, the difference between distortion-aware estimation and pinhole estimation is that distortion-aware estimation uses a calibration method that includes a fisheye distortion model, which can directly handle strong nonlinear distortion across the entire image range; while pinhole estimation is based on the pinhole model assumption and uses the characteristic that fisheye cameras approximately satisfy the pinhole model in the paraxial region to estimate extrinsic parameters and initial focal length.

[0083] The distortion-aware estimation method employs a calibration approach that incorporates a fisheye distortion model (e.g., a calibration method based on a polynomial fisheye model or an isometric projection model). It uses all corner points obtained in step S101 (including corner points at the center of the image and corner points in strongly distorted areas at the image edges) to jointly estimate the focal length and extrinsic parameters for each frame. Because this method directly models fisheye distortion during the estimation process, it can fully utilize corner constraints in edge regions and has good coverage of the distortion characteristics of the entire field of view of a fisheye lens. However, the accuracy of corner detection in edge regions is significantly affected by distortion; if the corner detection noise is high, the estimation results obtained by this method may produce corresponding deviations.

[0084] The pinhole estimation method is based on the pinhole model assumption, utilizing the characteristic that fisheye cameras have minimal distortion in the paraxial region (a small field of view near the center of the image), which approximates an ideal pinhole camera, to estimate extrinsic parameters and initial focal length. The pinhole estimation method further includes two sub-methods: The first sub-method is central region pinhole estimation: only corner points located in the central region of the calibration image are selected, and extrinsic parameters are estimated using a pinhole model (e.g., the PnP algorithm) to obtain the initial focal length. This method only utilizes corner points with minimal distortion in the central region of the image, excluding corner points with strong edge distortion, thus the estimated extrinsic parameters are relatively robust. However, this method does not utilize corner point information in the edge region and is insensitive to situations where the calibration plate covers the edge of the image. Here, the central region is the area centered on the principal point of the calibration image and covering a preset proportion of the total number of pixels in the calibration image.

[0085] The second sub-method is global pinhole estimation: using all corner points of the calibrated image (including corner points in the central and edge regions), the extrinsic parameters are estimated using the pinhole model to obtain the initial focal length. This method utilizes all corner point data, but due to strong distortion in the corner points of the edge regions, their projection relationship does not satisfy the pinhole model assumption. Therefore, more outliers are introduced, which may affect the pose estimation accuracy to some extent.

[0086] The three pose estimation methods (distortion-aware estimation, central region pinhole estimation, and global pinhole estimation) have different structures and different bias characteristics: distortion-aware estimation has strong coverage of edge distortion, but may introduce bias when the noise of edge corner detection is large; central region pinhole estimation uses the central region corner with the least distortion, and the pose result is more robust, but it does not utilize edge distortion information; global pinhole estimation utilizes all corner data, but the strong distortion of edge corners makes it not fully satisfy the pinhole model assumption, and may have systematic errors. The three methods have different bias directions. By running the three methods in parallel to obtain multiple sets of initial estimation results, the probability that at least one set of initial estimation results falls within the correct parameter region can be effectively increased. It should be noted that the above three estimation methods are only examples. In other implementations, any two of the above three methods can be used, or other structurally different estimators (such as estimators based on lines / vanishing points, or estimators based on known approximate intrinsic parameters) can be added, all of which fall within the protection scope of this disclosure.

[0087] In some embodiments, determining a set of candidate calibration parameters for each combination result includes: for each combination result, performing a scaling transformation on the initial focal length based on the model parameter values ​​in the combination result to obtain in-focal-length parameter values; the scaling transformation is used to make the equivalent focal length of the imaging model in the paraxial region equal to the initial focal length; and using the in-focal-length parameter values, the model parameter values, preset initial principal point values, preset initial distortion coefficient values, and the extrinsic parameters in the initial estimation result as the candidate calibration parameters.

[0088] For example, since the initial focal length given by the pose estimator is based on a certain scale reference under the pinhole model assumption or the fisheye model, and the focal length component of the UCM model is related to the model parameters... There is a coupling relationship; directly assigning this initial focal length to the focal length component of the UCM will cause the projection scale implied by the intrinsic parameters to be inconsistent with the geometric scale upon which the extrinsic parameters are based. Therefore, for each set of initial estimation results obtained in step S102, the initial focal length... With multiple preset model parameter values ​​(i.e., the unified model parameters in the UCM model) After combining the multiple possible values, for each Combine them to determine a complete set of candidate calibration parameters in the following manner.

[0089] First, based on the model parameter values ​​in the current combination result. A scaling transformation is performed on the initial focal length f to obtain the in-focal-length parameter values, including the lateral equivalent focal length. and longitudinal equivalent focal length (During the initial value construction phase, in order to simplify and maintain isotropy, we can let) This scaling transformation is used to make the equivalent focal length of the imaging model in the paraxial region equal to the initial focal length. That is, to make the projection scale of the imaging model, determined by the initial focal length and the model parameter values, consistent with the scale of the pose (the initial focal length represents the scale of the pose). As mentioned above, the equivalent focal length of the UCM model in the paraxial region... To ensure that the projection scale of the constructed UCM intrinsic parameters in the paraxial region is consistent with the scale used for pose estimation, it is necessary to make Thus obtain This transformation is essentially multiplying the initial focal length by a scale factor. This scaling factor is determined by the unified model parameters. Decide. The larger the value, the greater the magnitude of the scale transformation.

[0090] The physical meaning of this scaling transformation can be understood as follows: the pinhole model estimator provides an initial focal length *f* when estimating extrinsic parameters, representing the equivalent focal length under the paraxial pinhole approximation. However, the focal length component of the UCM model... It is not directly equal to the equivalent focal length, but rather determined by... and This is jointly determined. Through the above scaling transformation, the initial focal length f is mapped to a scale that is consistent with the current focal length. Matching UCM focal length components This allows both to achieve self-consistency in the projection scale.

[0091] After determining the parameter values ​​within the focal length Next, the focal length parameter values ​​and the model parameter values ​​in the current combination are used. The preset initial values ​​of the principal points, the preset initial values ​​of the distortion coefficients, and the extrinsic parameters from this initial estimation result are collectively used as a set of candidate calibration parameters. The initial value of the principal points is typically taken as the geometric center of the image, i.e., , ; Image width dimension This represents the image height dimension. The radial distortion coefficients are initially set to zero, i.e. , The principal point and distortion coefficient are set to the aforementioned preset values ​​in the initial stage, and will be used as variables to be optimized in the subsequent nonlinear optimization process.

[0092] At this point, a complete set of candidate calibration parameters includes: in-focal-length parameter values ​​determined through scaling. , Preset initial values ​​of principal points , (For example, the initial value of the principal point is the coordinates of the geometric center of the calibration image), and the unified model parameter values ​​in the current combination. Preset initial value of distortion coefficient , (For example, the initial value of the distortion coefficients is zero), and the extrinsic parameters (rotation and translation) of each frame in the initial estimation result. Repeat the above process, traversing all initial estimation results and all presets. By combining values, multiple sets of candidate calibration parameters can be generated.

[0093] In some embodiments, the camera to be calibrated is a fisheye camera; the unified camera model is superimposed with radial distortion to describe the projection relationship of the fisheye camera; the model parameters are the geometric control parameters of the unified camera model.

[0094] For example, the camera to be calibrated is a fisheye camera. Fisheye cameras, employing ultra-wide-angle fisheye lenses, possess an extremely large field of view (typically reaching 180° or even greater), enabling them to capture a wider range of scene information in a single image. Therefore, they are widely used in fields such as robot navigation, autonomous driving surround perception, and panoramic imaging for security monitoring. However, the ultra-wide-angle imaging of fisheye lenses introduces extremely strong nonlinear distortion, making it impossible for traditional pinhole models to accurately describe the imaging process. Therefore, a unified camera model (UCM) suitable for large field-of-view imaging is needed to model the projection relationship of the fisheye camera.

[0095] The imaging process of a fisheye camera comprises two core components: first, the principal projection relation (UCM), controlled by a unified camera model, describes the nonlinear mapping from 3D spatial points to the normalized image plane; second, a radial distortion correction term superimposed on the normalized image plane, used to compensate for residual distortion. In other words, the UCM describes the principal projection geometry of the fisheye camera (i.e., the mapping relationship between spatial points and imaging rays), while the radial distortion term corrects the residuals. Together, they constitute the complete projection model of the fisheye camera.

[0096] The model parameters in the unified camera model refer to the geometric control parameters of the unified camera model. . This is a crucial parameter in the UCM model, controlling the degree of distortion during the projection from a point in three-dimensional space onto the normalized image plane. Its physical meaning can be understood as a representation of the lens's field of view. The closer it is to 0, the closer the camera is to the pinhole model (i.e., an ideal projection with almost no distortion). The larger the value (theoretically ranging from [0, 1)), the larger the field of view and the stronger the distortion of the fisheye lens. Therefore, when constructing candidate calibration parameters, multiple different preset parameters are used. Values ​​(e.g., a range from 0 to close to 1) can cover various distortion levels from near-pinhole to strong fisheye, thus enabling the calibration method to adapt to the calibration needs of different field-of-view cameras and different types of fisheye cameras. Meanwhile, in subsequent steps... It will be used as one of the seven intrinsic parameters in nonlinear optimization to solve for the specific value that most accurately describes the projection behavior of the fisheye camera to be calibrated.

[0097] In some embodiments, the plurality of preset model parameter values ​​include a plurality of preset distortion degree parameter values; the plurality of preset distortion degree parameter values ​​are arranged in ascending or descending order and cover the effective value range of the unified camera model to characterize different degrees of fisheye distortion.

[0098] For example, the multiple preset model parameter values ​​include multiple preset distortion degree parameter values, that is, geometric control parameters in the unified camera model. Multiple possible values. These preset values... The values ​​are arranged in ascending or descending order and cover the effective range of the UCM model to represent different degrees of fisheye distortion from weak to strong (or from strong to weak).

[0099] Specifically, in the UCM model The valid range of values ​​is usually [0, 1). When When = 0, the UCM model degenerates into an ideal pinhole model, representing the distortion-free case; when When the value approaches 1, it indicates the strongest fisheye distortion and the largest field of view. Based on this range, a set of values ​​arranged in ascending order can be preset. Seed value, for example This covers the entire range from near pinhole to strong distortion. Alternatively, it can be arranged in descending order, for example, from 0.95 to 0.

[0100] Because different models of fisheye lenses have significantly different field of view and distortion characteristics, the preset... The values ​​are not limited to the examples above. In specific implementations, the values ​​can be determined based on the prior information of the field of view of the fisheye lens to be calibrated. The seed value range can be adaptively adjusted. For example, for fisheye lenses with a small field of view, the seed can be concentrated in a smaller value range (e.g., 0~0.5); for fisheye lenses with a large field of view, the seed can be expanded to a range closer to 1 (e.g., 0.5~0.95). When prior information is lacking, several values ​​can also be sampled evenly within the effective value range of the UCM model to fully cover possible values. Value space.

[0101] The aforementioned multiple preset unified model parameter seeds are not limited to a fixed set. In other embodiments, the multiple preset model parameter values ​​can also adaptively determine their value ranges based on the prior information of the field of view of the camera to be calibrated. For example, based on the size of the nominal field of view of the lens, the seeds can be generated concentrated within the corresponding range; or a two-level grid strategy of coarse sampling followed by fine sampling can be adopted, that is, initial candidates are generated by coarse sampling within the effective value range with a larger step size, and after preliminary scoring and sorting, secondary fine sampling is performed near the seed with the best score with a smaller step size, thereby increasing the seed density in key areas while controlling the total number of candidates.

[0102] By assigning multiple presets to the initial focal length f in each set of initial estimation results. Seed values ​​can generate multiple candidate calibration parameters covering different degrees of distortion. When the true degree of distortion of the fisheye lens to be calibrated is unknown, multiple preset values ​​can be used. Seeds can cover different distortion levels of hypotheses, ensuring that there are at least some candidates. The initial values ​​are close to the true values, so in the subsequent reprojection error sorting and optimization, the calibration results that best fit the observation data under the assumption of correct distortion degree can be selected, which further improves the adaptability of the calibration method to different types of fisheye cameras.

[0103] The above embodiments primarily address the calibration of monocular fisheye cameras. In practical applications, fisheye cameras may also be deployed on robots or autonomous vehicles in the form of multi-view systems (such as binocular stereo vision systems, surround-view quad-view systems, etc.). In this case, in addition to the accurate intrinsic parameters of each camera, the relative extrinsic parameters between the cameras can also be determined to achieve spatial fusion of multi-view perception information. To meet this requirement, in some embodiments, the above method is also extended to multi-view fisheye systems containing at least two fisheye cameras.

[0104] In some embodiments, the method further includes: applying the method to a multi-eye fisheye system containing at least two fisheye cameras, executing the method on each fisheye camera in the multi-eye fisheye system to obtain the intrinsic and extrinsic parameters of each fisheye camera; utilizing the inter-eye co-view constraint formed by the same calibration target appearing simultaneously in the fields of view of at least two fisheye cameras at the same time, and using the intrinsic and extrinsic parameters of each fisheye camera as initial values, performing joint global optimization on the intrinsic parameters of all fisheye cameras and the relative extrinsic parameters between the cameras.

[0105] For example, the camera calibration method described above can be applied to multi-eye fisheye systems (e.g., binocular fisheye systems, surround-view quad-eye systems, etc.) that include at least two fisheye cameras. Multi-eye systems are typically deployed in autonomous vehicles or robotic platforms, using multiple fisheye cameras to cover different fields of view, collectively achieving a wide-range seamless perception of the surrounding environment. In multi-eye system calibration, in addition to solving the intrinsic parameters of each camera, the relative extrinsic parameters between the cameras (i.e., the rotation and translation relationships between the coordinate systems of each camera) can also be determined to ensure that the information perceived by multiple cameras can be fused under the same spatial reference. The multi-eye calibration implementation method of this disclosure includes the following two main stages.

[0106] In the first stage, each eye is calibrated independently. For example, for each fisheye camera in a multi-eye fisheye system, the aforementioned steps S101 to S104 are executed independently. That is, each camera acquires a calibration target image, generates multi-source candidate initial values, performs unified scoring and ranking, optimizes and stabilizes the multi-candidate data, and finally obtains the intrinsic parameters (including...) of each eye camera. , , , , , , The calibration process involves both intrinsic parameters (i.e., the pose of the camera relative to the calibration target during each frame acquisition) and extrinsic parameters (i.e., the pose of the camera relative to the calibration target during each frame acquisition). The calibration processes for each camera are independent and do not affect each other; they can be executed serially or in parallel to improve calibration efficiency. Through this stage, each camera obtains a set of intrinsic parameters that accurately describe its own projection characteristics, as well as extrinsic parameters for each frame acquired by that camera.

[0107] In the second stage, joint global optimization between cameras can be performed. After the independent calibration of each camera is completed, the calibration accuracy can be further optimized by leveraging the characteristic that all cameras in a multi-camera system share the same calibration target for image acquisition. For example, during the calibration data acquisition stage, images can be acquired simultaneously by all cameras. That is, at the same moment, the same calibration target appears in the fields of view of two or more fisheye cameras. This forms an inter-camera co-view constraint: the projection observation of the same calibration target in images from different cameras at the same moment is actually an independent observation of the same calibration target in the coordinate systems of each camera under the same spatial pose. These observations are intrinsically geometrically linked through the extrinsic parameters of each camera relative to the calibration target. These extrinsic parameters should satisfy the rigid transformation consistency determined by the relative pose relationship between the multiple cameras. Therefore, using these co-view observations, a joint optimization objective function can be constructed, which includes the reprojection error of each camera in each frame, as well as the constraint terms introduced by the rigid transformation consistency between the extrinsic parameters of each camera. After completing the independent calibration of each camera, the intrinsic and extrinsic parameters obtained from the independent calibration of each camera are used as the initial values ​​for joint optimization. Further global refinement is then performed using inter-camera common-view constraints. That is, the intrinsic and extrinsic parameters obtained from the first stage of independent calibration of each camera are used as the initial values ​​for this joint optimization problem. Joint global optimization is then performed on the intrinsic parameters of all fisheye cameras and the relative extrinsic parameters between cameras to minimize the overall reprojection error of all cameras under shared-view observation. This multi-camera joint calibration, through the transfer and sharing of calibration information among cameras via common-view constraints, can suppress individual biases that may exist in single-camera calibration, making the relative poses between cameras more accurate. The final output of the multi-camera system's intrinsic and extrinsic parameter accuracy is superior to the results of independent calibration of each camera.

[0108] In the above embodiments, the multi-candidate strategy can, to some extent, compensate for the sensitivity to initial values. However, in actual calibration operations, if the acquired calibration images lack sufficient perspective diversity in the pose of the calibration targets (e.g., all calibration targets are facing the camera and concentrated in the center of the image), the constraints of the observation data on certain intrinsic parameter degrees of freedom (such as edge distortion coefficients, principal points, etc.) may not be sufficient, thus affecting the accuracy of the calibration results. To provide more adequate assurance of calibration quality during the data acquisition stage, in some embodiments, a real-time data quality assessment and active guidance mechanism can be introduced during the calibration image acquisition process.

[0109] In some embodiments, the method further includes: during the acquisition of the multi-frame calibration images, evaluating the spatial distribution of the extrinsic parameters of the calibration target in each frame of calibration images based on the image coordinates of corner points and their corresponding spatial coordinates in the acquired calibration images, and statistically analyzing the degree of constraint of the acquired calibration images in each degree of freedom. When the degree of constraint of any degree of freedom is lower than a preset threshold, guiding information is generated, which prompts the operator to adjust the orientation of the calibration target and acquire additional calibration image frames. After acquiring the additional calibration image frames acquired based on the guiding information, the step of acquiring the image coordinates of each corner point and their corresponding spatial coordinates in the multi-frame calibration images captured by the camera to be calibrated continues.

[0110] For example, during the acquisition of multiple calibration images, a real-time evaluation and proactive guidance mechanism for the quality of the calibration data is added. The core purpose of this mechanism is to ensure that the acquired calibration images can adequately constrain each degree of freedom of the calibration problem, avoiding a decrease in calibration accuracy or even failure due to a lack of perspective diversity in the observation data.

[0111] During the calibration image acquisition process, after each new calibration image frame is added, the spatial distribution of the extrinsic parameters of the calibration target in each frame is evaluated based on the image coordinates of the corner points and their corresponding spatial coordinates in all currently acquired calibration images. Specifically, for each acquired calibration image frame, the pose (rotation matrix and translation vector) of the calibration target relative to the camera can be calculated using existing pose estimation methods (such as the PnP algorithm), thus obtaining the orientation and position of the calibration target in the camera coordinate system. Based on this, the spatial distribution of the calibration target pose in all acquired frames is statistically analyzed, such as the coverage of the calibration target normal direction (distribution of pitch and yaw angles), the frequency of the calibration target appearing in different areas of the image, and the degree of tilt of the calibration target plane relative to the image plane. Based on these distribution statistics, the degree of constraint of the acquired data on each degree of freedom in the calibration problem (including the focal length, principal point, distortion coefficient, and ξ in the intrinsic parameters, and the rotation and translation components in the extrinsic parameters) can be quantitatively evaluated. For example, if the calibration target is always facing the camera and centered in all acquired calibration target images, there is a lack of constraints on the degrees of freedom corresponding to the tilt posture of the calibration target; if the calibration target always appears on the same side of the image, there is a lack of constraints on the distortion of the edge region on the other side. The degree of these constraints can be measured using quantitative indicators, such as the coverage of the calibration target normal direction in the pitch angle dimension and the variance of the distribution of the calibration target center on the image plane.

[0112] When the system detects that the constraint level of any degree of freedom is lower than a preset threshold, it can automatically generate guidance information to prompt the operator to adjust the orientation of the calibration target and acquire additional calibration image frames. For example, if the system detects that the pitch angle variation range of the calibration target in the acquired images is too small (i.e., all calibration targets are approximately facing the camera directly, lacking tilt posture), it generates guidance information prompting "Please tilt the calibration target at a certain angle (e.g., pitch 30°) and acquire another set of images"; if the system detects that the position distribution of the calibration targets in the image is too concentrated, lacking coverage of the edge areas of the field of view, it generates guidance information prompting "Please move the calibration target to the edge areas such as the upper left / upper right corner of the image and acquire another set of images". Through the above guidance information, the operator can obtain clear guidance on the acquisition direction during the calibration process, without having to rely on their own experience to judge whether the data is sufficient. After the operator acquires new calibration image frames according to the guidance information, step S101 is executed again to include the newly acquired image frames in the acquired dataset and continue to evaluate the constraint level. This process is repeated until the constraints on all degrees of freedom reach the preset threshold, indicating that the collected calibration data has sufficient perspective diversity to adequately constrain the calibration problem. Then, subsequent steps such as candidate generation, scoring and ranking, and optimization are performed.

[0113] By adopting this implementation method, the quality of calibration input data can be ensured, the effectiveness of the multi-candidate strategy can be maximized, and the calibration accuracy or failure due to poor input data quality can be avoided.

[0114] The above embodiments, through steps such as multi-candidate generation, unified scoring and ranking, and two-stage optimization, ultimately output a set of calibration parameters. In practical applications, operators may find it difficult to judge the reliability of the calibration result based solely on the parameters themselves, especially when lacking professional experience, and may be unable to identify whether the calibration is biased due to insufficient data quality or outlier interference. Therefore, in some embodiments, the above methods further introduce a quantitative evaluation and diagnostic assistance mechanism for calibration quality after outputting the calibration results.

[0115] In some embodiments, the method further includes: after obtaining the target calibration parameters, calculating the reprojection error for all corner points used for calibration and statistically analyzing the distribution of the reprojection error to form an error distribution statistical result; calculating the parameter covariance matrix based on the objective function value at the convergence of the nonlinear optimization, and obtaining the confidence interval of each parameter in the calibration result from the parameter covariance matrix; weighting the error distribution statistical result with the confidence interval of each parameter to obtain a calibration confidence score; and generating diagnostic prompt information when the calibration confidence score is lower than a preset score threshold, the diagnostic prompt information including at least one of the following: when the width of the confidence interval of a parameter exceeds a preset width threshold, prompting to increase the observation in the sensitive direction of the parameter; when the deviation of the high quantile from the mean in the error distribution statistical result exceeds a preset deviation threshold, prompting to check the corner detection quality; and when the mean of the error distribution statistical result exceeds a preset mean threshold, prompting to add calibration target images with different orientations.

[0116] For example, after obtaining the target calibration parameters, the reliability of the calibration results is quantitatively evaluated, and diagnostic prompts are output to the operator based on the evaluation results. Operators can determine the reliability of the calibration results without relying on professional experience. This specifically includes the following aspects: Firstly, reprojection error distribution statistics are performed. After obtaining the target calibration parameters, these parameters are substituted into the UCM imaging model. The reprojection error is calculated for all corner points used for calibration (including corner points previously marked as outliers during nonlinear optimization). The distribution statistics of these reprojection errors are then statistically analyzed to form the error distribution statistics. These statistics include at least one or more of the following: mean error, standard deviation, maximum value, and 90th percentile. Optionally, a histogram of the error distribution can also be output to visually demonstrate the error distribution pattern. If the mean error differs significantly from the 90th percentile (e.g., the 90th percentile is significantly higher than the mean), it indicates the existence of a small number of corner points with extremely large errors. These corner points may be introduced due to corner point misdetection, target occlusion, or reflection, interfering with the calibration results.

[0117] Secondly, parameter confidence intervals are evaluated. Based on the loss function value at the convergence point of the nonlinear optimization, the covariance matrix of the calibration parameters near the convergence point is calculated. The basic calculation logic is as follows: At the convergence point, the objective function (i.e., the loss function) is reconciled with respect to the calibration parameter vector: The Hessian matrix H is obtained by estimating the second derivatives of the extrinsic parameters for each frame. This matrix reflects the local curvature information of the objective function at the convergence point. The inverse of the Hessian matrix is ​​an approximate estimate of the covariance matrix of the calibration parameters. ,in, The covariance matrix of the calibration parameters; The Hessian matrix is ​​the objective function. At the convergence point, with respect to the calibration parameter vector The second-order partial derivative matrix of the covariance matrix. The square root of the diagonal elements of this covariance matrix is ​​the estimate of the standard deviation of each parameter. Therefore, the parameters The confidence interval at a given confidence level can be approximately expressed as: ,in, For calibration parameter vector The first in parameters (e.g.) , , (and so on) For the first The standard deviation estimates of the parameters, The parameter is the covariance matrix; For the standard normal distribution quantiles, The significance level. For example, at a 95% confidence level, , In one specific implementation, the Hessian matrix can be obtained by numerically differencing the loss function at the optimization convergence point, or it can be automatically calculated and output by a nonlinear optimizer (such as Ceres Solver) during the optimization process.

[0118] The above method can be used to obtain the various parameters in the calibration results, including the focal length component. , Main point , Unified model parameters and second-order radial distortion coefficient , The confidence interval of a parameter is an important indicator of the reliability of the calibration results. A wider confidence interval for a parameter indicates a gentler curvature of the objective function in that parameter's direction, meaning the parameter is less constrained by the currently acquired image data. This suggests a lack of observational information sensitive to this parameter in the calibration image, making it impossible to determine the parameter precisely. This implies that if images are reacquired for calibration, the parameter's fluctuations may be significant. Conversely, a narrower confidence interval indicates a tighter constraint on the parameter by the observed data, resulting in a more stable and reliable calibration result. For example, if the principal point... An excessively wide confidence interval means that the distribution of the calibration target in the horizontal direction of the image is insufficient, and the observation information on the left and right sides of the image is insufficient, resulting in the inability to accurately determine the horizontal position of the principal point.

[0119] Thirdly, a comprehensive confidence score and decision support are provided. The above error distribution statistics and the confidence intervals of each parameter are weighted to obtain a calibration confidence score. This score can be a quantitative value from 0 to 100, comprehensively reflecting the reliability of the calibration results. When the confidence score is lower than a preset threshold, a diagnostic prompt is automatically generated. The specific content of the diagnostic prompt can vary depending on the specific anomaly. For example, when the width of the confidence interval for a parameter exceeds a preset width threshold, the prompt suggests "Increase observations in the sensitive direction of this parameter." For instance, if the principal point confidence interval is too wide, it suggests "Increase observations of the calibration target at the edge of the image"; if the focal length confidence interval is too wide, it suggests "Increase observations of the calibration target at different depths." Furthermore, when the deviation between the high quantile (e.g., the 90th percentile) and the mean in the error distribution statistics exceeds a preset deviation threshold, the prompt suggests "There are many outliers; it is recommended to check the corner detection quality or whether the calibration target has occlusion / reflection." This prompt points to data quality issues, helping operators to check for systematic errors in the corner detection process. For example, when the mean of the error distribution statistics exceeds a preset mean threshold, a message appears stating, "Insufficient calibration accuracy; it is recommended to add more calibration target images with different orientations." This message points to the inadequacy of the observation data in terms of visual diversity, guiding the operator to collect more calibration images with varying attitudes.

[0120] By employing the above implementation method, a comprehensive reliability evaluation of the calibration results is provided through a three-tiered progressive analysis of error distribution statistics, parameter confidence interval assessment, and comprehensive confidence score. When the calibration results are unreliable, targeted diagnostic prompts and improvement suggestions can be provided, enabling operators to determine the usability of the calibration results without relying on professional experience, and to obtain clear directions for improvement when the calibration accuracy is insufficient.

[0121] The camera calibration method provided in this disclosure, when applied to a fisheye camera, takes images acquired by the physical fisheye camera in an environment containing multiple non-coplanar calibration targets as input. After extracting the two-dimensional pixel observations of corner points and their corresponding three-dimensional points, it sequentially processes the data through five core modules: observation acquisition, multi-source candidate generation, unified reprojection scoring and sorting, multi-candidate optimization, and stable and optimal output. The final output is the seven-dimensional intrinsic parameters of the UCM and the extrinsic parameters of each frame. Each processing step is described in module form. The observation acquisition module is used to acquire images from the fisheye camera in an environment containing multiple non-coplanar calibration targets, detect corner points on the calibration targets, and obtain the corresponding set of two-dimensional pixel observations and three-dimensional spatial points for each frame. The candidate generation module contains at least two structurally distinct pose estimation sub-modules used to obtain pose and focal length estimates in parallel. The unified model parameter seed configuration sub-module is used to provide multiple preset model parameter values. The focal length intrinsic parameter self-consistent determination sub-module is used to construct complete UCM initial value candidates in a manner that allows the focal length to participate in the autonomous path pose scale. The unified UCM reprojection scoring and ranking module calculates and ranks all candidates under the unified UCM (including radial distortion) projection relationship, calculating reprojection errors. The multi-candidate two-stage optimization module independently performs two-stage nonlinear optimization on the top-ranked candidates. The stability selection and output module uses the average reprojection error of all participating points for stability selection, stopping early when the target is met and outputting the calibration results. These modules are interconnected and work collaboratively to complete the entire process from original image input to calibration parameter output. By combining multi-source heterogeneous pose estimation with multiple preset seeds to generate a broad set of candidate initial values, the probability of at least one set of candidates falling into the correct parameter region is fundamentally improved. The self-consistent determination of the scale of participating poses within the focal length eliminates systematic deviations between intrinsic and extrinsic initial values. Scoring and ranking under the unified UCM reprojection metric makes candidates with different structures comparable and selectable. Multi-candidate independent optimization and stability selection using the average error of all points balance the global optimal probability with resistance to external interference. By employing a mechanism that stops once the calibration is achieved, the computation time is effectively controlled while ensuring calibration accuracy. The overall solution, with its tightly integrated modular architecture, collaboratively addresses issues such as initial value sensitivity, scale misalignment, and difficulty in selecting the optimal value in UCM fisheye calibration, significantly reducing the calibration failure rate and improving the stability and reproducibility of the calibration results.

[0122] The calibration results obtained in this embodiment include the seven-dimensional intrinsic parameters of UCM and the extrinsic parameters of each frame. These can be directly used to perform distortion correction and rectification on the images acquired by the fisheye camera to obtain the distortion-corrected images for subsequent visual tasks. They can also be used for multi-sensor fusion perception of robots or mobile platforms, such as spatially aligning and fusing the corrected fisheye images with data from other sensors such as LiDAR and inertial measurement units.

[0123] like Figure 3The diagram shown is a schematic representation of a camera calibration apparatus 300 provided in an exemplary embodiment of this disclosure. The camera calibration apparatus includes: The acquisition module 31 is used to acquire the image coordinates of each corner point and its corresponding spatial coordinates in multiple frames of calibration images captured by the camera to be calibrated; the imaging model of the camera to be calibrated is a unified camera model. The initial estimation module 32 is used to estimate the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration image and the initial focal length of the camera to be calibrated, respectively, using a variety of different pose estimation methods, based on the image coordinates and the spatial coordinates, to obtain multiple sets of initial estimation results; The candidate calibration module 33 is used to combine the initial focal length in each group of initial estimation results with multiple preset model parameter values, and determine a set of candidate calibration parameters for each combination result, with the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length. The target calibration module 34 is used to substitute each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point, and select the target calibration parameter as the calibration result from each set of candidate calibration parameters based on the reprojection error corresponding to each set of candidate calibration parameters.

[0124] In one embodiment, the target calibration module 34 selects a target calibration parameter as the calibration result from each set of candidate calibration parameters based on the reprojection error corresponding to each set of candidate calibration parameters. This includes: determining the reprojection error statistics corresponding to each set of candidate calibration parameters based on the reprojection error of each corner point corresponding to each set of candidate calibration parameters; selecting a preset number of sets of candidate calibration parameters based on the reprojection error statistics corresponding to each set of candidate calibration parameters; performing nonlinear optimization independently on each set of candidate calibration parameters; and selecting a target calibration parameter as the calibration result from the preset number of sets of candidate calibration parameters based on the reprojection error statistics of each corner point corresponding to each set of candidate calibration parameters after optimization.

[0125] In one implementation, the target calibration module 34 independently performs nonlinear optimization for each selected set of candidate calibration parameters, including: in a first stage, using a squared loss function as the optimization objective, jointly optimizing the intrinsic parameters of the candidate calibration parameters and the extrinsic parameters of each frame of calibration image until the parameter change is less than a preset threshold; in a second stage, the optimization objective is switched to a Cauchy bar kernel, and in each iteration, corner points whose single-point reprojection error exceeds a preset multiple of the standard deviation are eliminated according to the statistical distribution of the reprojection error of all current corner points, and the intrinsic and extrinsic parameters of the candidate calibration parameters are continued to be jointly optimized; the intrinsic and extrinsic parameters of the candidate calibration parameters after optimization in the first and second stages are output.

[0126] In one embodiment, the target calibration module 34 selects a preset number of candidate calibration parameters based on the reprojection error statistics corresponding to each group of candidate calibration parameters, and independently performs nonlinear optimization for each selected group of candidate calibration parameters. This includes: sorting the candidate calibration parameters corresponding to the multiple initial estimation results in ascending order of the reprojection error statistics corresponding to each group of candidate calibration parameters; and, based on the sorting results, sequentially selecting candidate calibration parameters in ascending order from the group with the smallest reprojection error statistics to perform the nonlinear optimization until the optimization of the preset number of candidate calibration parameters is completed. Wherein, when the reprojection error statistics of any group of candidate calibration parameters after optimization is less than or equal to a preset threshold, the optimization of subsequent groups of candidate calibration parameters is stopped, and the optimized result of that group of candidate calibration parameters is output as the target calibration parameter.

[0127] In one embodiment, the various pose estimation methods include a distortion-aware estimation method and a pinhole estimation method; the distortion-aware estimation method includes: using a fisheye distortion model and estimating the extrinsic parameters and the initial focal length using all corner points; the pinhole estimation method includes a first sub-method and a second sub-method; the first sub-method includes: using only corner points located in the center region of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model; the second sub-method includes: using all corner points of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model.

[0128] In one embodiment, the candidate calibration module 33 determines a set of candidate calibration parameters for each combination result, including: for each combination result, performing a scaling transformation on the initial focal length according to the model parameter values ​​in the combination result to obtain in-focal-length parameter values; the scaling transformation is used to make the equivalent focal length of the imaging model in the paraxial region equal to the initial focal length; and using the in-focal-length parameter values, the model parameter values, the preset initial principal point values, the preset initial distortion coefficient values, and the extrinsic parameters in the initial estimation result as the candidate calibration parameters.

[0129] In one embodiment, the camera to be calibrated is a fisheye camera; the unified camera model is superimposed with radial distortion to describe the projection relationship of the fisheye camera; the model parameters are the geometric control parameters of the unified camera model.

[0130] In one embodiment, the plurality of preset model parameter values ​​include a plurality of preset distortion degree parameter values; the plurality of preset distortion degree parameter values ​​are arranged in ascending or descending order and cover the effective value range of the unified camera model to characterize different degrees of fisheye distortion.

[0131] In one embodiment, the camera calibration device 300 may further include a joint optimization module 35, used to control the execution of the above modules to calibrate a multi-eye fisheye system containing at least two fisheye cameras; wherein, for each fisheye camera in the multi-eye fisheye system, the functions of the above modules are run respectively to obtain the intrinsic and extrinsic parameters of each fisheye camera; using the inter-eye co-view constraint formed by the same calibration target appearing in the field of view of at least two fisheye cameras at the same time, the intrinsic and extrinsic parameters of each fisheye camera are used as initial values ​​to jointly optimize the intrinsic parameters of all fisheye cameras and the relative extrinsic parameters between the cameras globally.

[0132] In one embodiment, the acquisition module 31 is further configured to, during the acquisition of the multi-frame calibration images, evaluate the spatial distribution of the extrinsic parameters of the calibration target in each frame of calibration images based on the image coordinates of the corner points and their corresponding spatial coordinates in the acquired calibration images, and statistically analyze the degree of constraint of the acquired calibration images in each degree of freedom; when the degree of constraint of any degree of freedom is lower than a preset threshold, generate guidance information, which is used to prompt the operator to adjust the orientation of the calibration target and supplement the acquisition of calibration image frames; after acquiring the calibration image frames supplemented according to the guidance information, continue to execute the step of acquiring the image coordinates of each corner point and their corresponding spatial coordinates in the multi-frame calibration images captured by the camera to be calibrated.

[0133] In one embodiment, the target calibration module 34 is further configured to: After obtaining the target calibration parameters, the reprojection error is calculated for all corner points used for calibration, and the distribution statistics of the reprojection error are statistically analyzed to form an error distribution statistical result. The parameter covariance matrix is ​​calculated based on the loss function value at the convergence of the nonlinear optimization, and the confidence interval of each parameter in the calibration result is obtained from the parameter covariance matrix. The error distribution statistical result and the confidence interval of each parameter are weighted to obtain a calibration confidence score. When the calibration confidence score is lower than a preset score threshold, a diagnostic prompt message is generated.

[0134] The detailed implementation process of each module of the aforementioned camera calibration device is essentially the same as that of the aforementioned method, and can be found in the description of the aforementioned method content, so it will not be repeated here.

[0135] Reference Figure 4The diagram shown is a structural schematic of a camera calibration device 400 provided in an embodiment of this disclosure. The camera calibration device 400 can be a computer device or a control chip, etc., and can be deployed on a server, a terminal, a robot, a remote control device, etc. It may include a processor 410, a memory 420, and a bus 430. The memory 420 is used to store execution instructions and includes a main memory 421 and an external memory 422. The main memory 421, also called internal memory, is used to temporarily store computational data in the processor 410 and data exchanged with external memory 422 such as a hard disk. The processor 410 exchanges data with the external memory 422 through the main memory 421.

[0136] In this embodiment, the memory 420 is specifically used to store application code executing the scheme of this disclosure, and its execution is controlled by the processor 410. That is, when the camera calibration device 400 is running, the processor 410 communicates with the memory 420 through the bus 430, or the processor 410 communicates with the memory 420 through other means, so that the processor 410 executes the application code stored in the memory 420, thereby executing the steps of the camera calibration method described in any of the foregoing embodiments. The memory 420 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 410 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor.

[0137] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the camera calibration method described in any of the above embodiments. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as DVD-ROM, DVD-RAM, DVD-RW, DVD+RW, CD-ROM, CD-RW, CD-RW, and MO (magneto-optical) storage media; and semiconductor storage media, such as flash memory, EEPROM, Dynamic Random Access Memory (DRAM), and Static Random Access Memory (SRAM).

[0138] The computer program can be written in various computer programming languages, including but not limited to C, C++, Python, and custom messages and services under the ROS framework. When the computer program is executed by the processor, it implements the various steps of the camera calibration method in the embodiments of this disclosure.

[0139] This disclosure also provides a computer program product storing a computer program. When executed by a processor, the computer program performs the steps of the camera calibration method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here. The computer program product can be implemented using hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0140] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and apparatuses described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A camera calibration method, characterized by, include: Obtain the image coordinates and corresponding spatial coordinates of each corner point in multiple frames of calibration images captured by the camera to be calibrated; Using various pose estimation methods, the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration image and the initial focal length of the camera to be calibrated are estimated based on the image coordinates and the spatial coordinates, resulting in multiple sets of initial estimation results; The initial focal lengths in each set of initial estimation results are combined with multiple preset model parameter values, and a set of candidate calibration parameters is determined for each combination result, with the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length. Substitute each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point, and select the target calibration parameter from each set of candidate calibration parameters as the calibration result based on the reprojection error corresponding to each set of candidate calibration parameters.

2. The method according to claim 1, characterized in that, The step of selecting the target calibration parameter as the calibration result from each group of candidate calibration parameters based on the reprojection error corresponding to each group of candidate calibration parameters includes: Based on the reprojection error of each corner point corresponding to each group of candidate calibration parameters, determine the statistical value of the reprojection error corresponding to each group of candidate calibration parameters. Based on the reprojection error statistics corresponding to each group of candidate calibration parameters, a preset number of candidate calibration parameters are selected, and nonlinear optimization is performed independently for each selected group of candidate calibration parameters. Based on the reprojection error statistics of each corner point corresponding to each group of candidate calibration parameters after optimization, the target calibration parameter is selected from the preset number of candidate calibration parameters as the calibration result.

3. The method according to claim 2, characterized in that, The step of independently performing nonlinear optimization for each selected set of candidate calibration parameters includes: In the first stage, the intrinsic parameters of the candidate calibration parameters are jointly optimized with the extrinsic parameters of each frame calibration image, using the squared loss function as the optimization objective, until the parameter change is less than a preset threshold. In the second stage, the optimization target is switched to the Cauchy bar kernel, and in each iteration, corner points whose single-point reprojection error exceeds a preset multiple of the standard deviation are eliminated based on the statistical distribution of the reprojection error of all current corner points. The intrinsic and extrinsic parameters of the candidate calibration parameters are then jointly optimized. Output the intrinsic and extrinsic parameters of the candidate calibration parameters after optimization in the first and second stages.

4. The method according to claim 2, characterized in that, The step involves selecting a preset number of candidate calibration parameters based on the reprojection error statistics corresponding to each group of candidate calibration parameters, and independently performing nonlinear optimization for each selected group of candidate calibration parameters, including: Based on the reprojection error statistics corresponding to each group of candidate calibration parameters, the candidate calibration parameters corresponding to the multiple initial estimation results are sorted in ascending order; Based on the sorting results, starting from the group with the smallest reprojection error statistics, candidate calibration parameters are selected in ascending order to perform the nonlinear optimization until the optimization of the preset number of candidate calibration parameters is completed. Specifically, when the reprojection error statistics after optimization of any set of candidate calibration parameters are less than or equal to a preset threshold, the optimization of subsequent sets of candidate calibration parameters is stopped, and the result of the optimization of that set of candidate calibration parameters is output as the target calibration parameter.

5. The method according to claim 1, characterized in that, The various pose estimation methods include distortion-aware estimation and pinhole estimation; The distortion sensing estimation method includes: using a fisheye distortion model and using all corner points to estimate the extrinsic parameters and the initial focal length; The pinhole estimation method includes a first sub-method and a second sub-method; The first sub-method includes: using only the corner points located in the center region of the calibration image to estimate the extrinsic parameters and obtain the initial focal length using a pinhole model; The second sub-method includes: using all corner points of the calibrated image, estimating the extrinsic parameters using a pinhole model to obtain the initial focal length.

6. The method according to claim 1, characterized in that, The step of determining a set of candidate calibration parameters for each combination result includes: For each combination result, a scaling transformation is performed on the initial focal length based on the model parameter values ​​in the combination result to obtain intrafocal parameter values; the scaling transformation is used to make the equivalent focal length of the imaging model in the paraxial region equal to the initial focal length; The focal length parameter value, the model parameter value, the preset initial principal point value, the preset initial distortion coefficient value, and the extrinsic parameter in the initial estimation result are used as the candidate calibration parameters.

7. The method according to claim 1, characterized in that, The camera to be calibrated is a fisheye camera; the imaging model of the camera to be calibrated is a unified camera model; the unified camera model is superimposed with radial distortion to describe the projection relationship of the fisheye camera; the model parameters are the geometric control parameters of the unified camera model.

8. The method according to claim 1, characterized in that, The multiple preset model parameter values ​​include multiple preset distortion degree parameter values; the multiple preset distortion degree parameter values ​​are arranged in ascending or descending order and cover the effective value range of the imaging model to characterize different degrees of fisheye distortion.

9. The method according to claim 1, characterized in that, The method further includes: The method is applied to a multi-eye fisheye system containing at least two fisheye cameras. The method is executed on each fisheye camera in the multi-eye fisheye system to obtain the intrinsic and extrinsic parameters of each fisheye camera. By utilizing the inter-eye co-view constraint formed by the same calibration target appearing simultaneously in the fields of view of at least two fisheye cameras at the same time, and using the intrinsic and extrinsic parameters of each fisheye camera as initial values, the intrinsic parameters of all fisheye cameras and the relative extrinsic parameters between the cameras are jointly optimized globally.

10. The method according to claim 1, characterized in that, The method further includes: During the acquisition of the multi-frame calibration images, the distribution of the external parameters of the calibration target in space in each frame calibration image is evaluated based on the image coordinates of the corner points and their corresponding spatial coordinates in the acquired calibration images, and the degree of constraint of the acquired calibration images in each degree of freedom is statistically analyzed. When the constraint level of any degree of freedom is lower than a preset threshold, guidance information is generated. The guidance information is used to prompt the operator to adjust the orientation of the calibration target and supplement the acquisition of calibration image frames. After acquiring the calibration image frames supplemented by the guidance information, the next step is to acquire the image coordinates of each corner point and its corresponding spatial coordinates in the multiple calibration images captured by the camera to be calibrated.

11. The method according to claim 1, characterized in that, The method further includes: After obtaining the target calibration parameters, the reprojection error is calculated for all corner points used for calibration, and the distribution statistics of the reprojection error are statistically analyzed to form the error distribution statistics. The parameter covariance matrix is ​​calculated based on the loss function value at the convergence of the nonlinear optimization, and the confidence interval of each parameter in the calibration result is obtained from the parameter covariance matrix. The error distribution statistics are weighted with the confidence intervals of each parameter to obtain the calibration confidence score; When the calibrated confidence score is lower than a preset score threshold, a diagnostic prompt message is generated.

12. A camera calibration device, characterized in that, include: The acquisition module is used to acquire the image coordinates of each corner point and its corresponding spatial coordinates in multiple frames of calibration images captured by the camera to be calibrated. The initial estimation module is used to estimate the extrinsic parameters of the camera to be calibrated relative to the calibration target in each frame of calibration image and the initial focal length of the camera to be calibrated, respectively, using a variety of different pose estimation methods, based on the image coordinates and the spatial coordinates, to obtain multiple sets of initial estimation results; The candidate calibration module is used to combine the initial focal length in each set of initial estimation results with multiple preset model parameter values, and determine a set of candidate calibration parameters for each combination result, with the constraint that the equivalent focal length of the imaging model in the paraxial region is equal to the initial focal length. The target calibration module is used to substitute each set of candidate calibration parameters into the imaging model to calculate the reprojection error of each corner point, and select the target calibration parameter as the calibration result from each set of candidate calibration parameters based on the reprojection error corresponding to each set of candidate calibration parameters.

13. A camera calibration device, characterized in that, It includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions to perform the steps of the camera calibration method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the camera calibration method as described in any one of claims 1 to 11.

Images