Staged robust camera parameter optimization method and device, equipment and medium

Abstract

The invention discloses a staged robust camera parameter optimization method, device, equipment and medium, and relates to the field of camera parameter optimizing.The method comprises the steps that parameters of a to-be-calibrated camera are obtained and initialized, and the parameters are organized in a unified parameter vector; the parameter vector is subjected to multi-stage nonlinear optimization, the parameter vector is divided into a plurality of parameter subsets through the preset parameter mask, the parameter subsets are sequentially activated and optimized in the multiple stages, and the input of the next optimization stage is the output result of the previous stage; and outputting a final camera parameter obtained through a plurality of stages of nonlinear optimization processes. According to the main technical scheme and effects, by introducing a staged optimization strategy, all parameters to be estimated are divided into a plurality of parameter subsets, and optimization is carried out in sequence in a staged mode; the result of the previous stage is used as an initial value in each stage, so that the problems of overlarge solution space, unstable convergence and the like caused by one-time full-parameter optimization are avoided; and the risk of falling into local optimum is obviously reduced.

Description

Technical Field

[0001] The present invention relates to the field of camera parameter optimization, and in particular to a staged robust camera parameter optimization method, apparatus, device and medium. Background Art

[0002] With the rapid development of machine and computer vision technologies, camera calibration, an essential and critical step in vision systems, has become increasingly important. Its accuracy directly impacts the performance of subsequent tasks, such as image processing and 3D reconstruction. Currently, the most widely used camera calibration method is based on the planar calibration method proposed by Zhang Zhengyou. This method uses a simple planar checkerboard as a calibration plane. The camera captures images from multiple viewpoints and calculates the camera's intrinsic and extrinsic parameters by solving the homography matrix between the image and the world coordinates.

[0003] Although Zhang Zhengyou's method greatly simplifies the calibration process and can further improve accuracy through the Levenberg-Marquardt (LM) nonlinear optimization algorithm, this optimization process typically incorporates all parameters to be estimated (including intrinsic and extrinsic parameters, and distortion coefficients of multiple orders) into a single optimization framework. Due to the high parameter dimensionality and complex solution space, this type of overall optimization approach is prone to falling into local optimality, unstable convergence, or even non-convergence, affecting the stability and accuracy of the calibration.

[0004] Furthermore, in actual data collection, due to uncontrollable factors such as image noise, corner extraction errors, and reflective interference, image point data inevitably contain outliers with large errors. Traditional LM algorithms, which aim to minimize squared error, struggle to suppress the impact of these outliers on the optimization results, causing the final calibration results to deviate from the true value and resulting in poor robustness. Furthermore, existing LM algorithms have a relatively crude strategy for adjusting the damping factor, typically relying solely on a fixed number of iterations as a termination condition. This inability to adaptively determine convergence compromises overall optimization efficiency.

[0005] In summary, the existing technology still has the following problems to be solved:

[0006] Optimizing all parameters at once results in an overly high dimension of the optimization space, which can easily lead to falling into local optimality and poor convergence.

[0007] The lack of an effective mechanism to suppress image outliers leads to low robustness of the optimization results;

[0008] The parameter update strategy lacks dynamic adjustment capabilities and the optimization efficiency is low. Summary of the Invention

[0009] Purpose of the invention: The purpose of the present invention is to solve the technical problems in the prior art and provide a phased robust camera parameter optimization method, device, equipment and medium.

[0010] One or more embodiments of this specification also relate to a staged robust camera parameter optimization device, an electronic device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0011] Technical solution:

[0012] In a first aspect, the present application proposes a phased robust camera parameter optimization method, comprising:

[0013] Obtain and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector;

[0014] Performing a multi-stage nonlinear optimization on the parameter vector, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence in multiple stages, with the input of the subsequent optimization stage being the output of the previous stage;

[0015] Outputs the final camera parameters obtained after multiple stages of nonlinear optimization process.

[0016] Preferably, the parameter vector includes an internal parameter part, a distortion coefficient part and an external parameter part;

[0017] The internal parameter part includes focal length parameters and principal point coordinates.

[0018] The distortion coefficient part includes basic radial distortion coefficient, tangential distortion coefficient and high-order radial distortion coefficient;

[0019] The external parameter part includes a rotation vector and a translation vector.

[0020] Preferably, obtaining and initializing the parameters of the camera to be calibrated includes:

[0021] Call Zhang Zhengyou calibration algorithm to obtain the intrinsic parameter part, distortion coefficient part and extrinsic parameter part according to the input three-dimensional point set of the world coordinate system and its corresponding two-dimensional point set of the image coordinate system.

[0022] Preferably, the preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence at different stages, and the input of the subsequent optimization stage is the output result of the previous stage, including:

[0023] The optimization parameters are adjacent in three stages, where:

[0024] In the first stage, a first parameter subset is constructed through a parameter mask, where the first parameter subset includes the focal length parameter and the principal point coordinates of the intrinsic parameter part, the basic radial distortion coefficient of the distortion coefficient part, and the rotation vector and translation vector of the extrinsic parameter part;

[0025] In the second stage, a second parameter subset is constructed through the parameter mask, wherein the second parameter subset includes a tangential distortion coefficient additionally added through the distortion coefficient part on the basis of the result of the first stage;

[0026] In the third stage, a third parameter subset is constructed through a parameter mask, wherein the third parameter subset includes a high-order radial distortion coefficient that is additionally added to the distortion coefficient part based on the result of the second stage.

[0027] Preferably, the nonlinear optimization process includes optimizing the Levenberg-Marquardt algorithm based on the trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, which includes:

[0028] The theoretical image point position is calculated based on the current parameter vector, compared with the actual image point to generate an error vector, and the partial derivatives of the error with respect to the activation parameters of the current parameter subset are solved to construct the Jacobian matrix;

[0029] The absolute value of the reprojection error of all data points is obtained from the error vector, the median absolute deviation is calculated based on the absolute value of the reprojection error of all data points, the standard deviation of the error distribution is estimated, and the Huber loss function is used to assign weights to each data point based on the standard deviation;

[0030] Multiplying the weight by the error vector element by element to obtain a weighted error vector, and multiplying the weight by the Jacobian matrix row by row to obtain a weighted Jacobian matrix;

[0031] The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size;

[0032] The gain ratio is calculated to evaluate the effectiveness of the update, and the gain ratio threshold is used to decide whether to accept the parameter update and adjust the damping factor.

[0033] Preferably, constructing and solving the Levenberg-Marquardt incremental equation based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size includes:

[0034] Substituting the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is as follows:

[0035] ;

[0036] in, is the transpose of the weighted Jacobian matrix, is the weighted Jacobian matrix, is the weighted error vector, is the damping factor, is the identity matrix, is the parameter update step size;

[0037] The approximate Hessian matrix JtJ and gradient g are calculated by the weighted Jacobian matrix J_w and the weighted error vector err_w. The formula is as follows;

[0038] ;

[0039] ;

[0040] By substituting the approximate Hessian matrix JtJ and the gradient g into the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is obtained:

[0041] ;

[0042] Get the parameter update step size, the formula is as follows:

[0043] .

[0044] Preferably, calculating the gain ratio to evaluate the update effectiveness, and deciding whether to accept the parameter update and adjusting the damping factor according to the gain ratio threshold, include:

[0045] The candidate parameter vector is calculated as follows:

[0046] ;

[0047] in, is the parameter vector subset of the current input;

[0048] The actual error reduction is calculated as follows:

[0049] ;

[0050] The actual error reduction is calculated as follows:

[0051] ;

[0052] The gain ratio is obtained as follows:

[0053] ;

[0054] The gain ratio is compared with the preset positive threshold. If it is greater than the positive threshold, it means that the update is valid, and the parameter vector subset is updated to the candidate parameter vector, while reducing the damping factor;

[0055] If the gain ratio is less than the positive threshold, no update is performed, and the damping factor is increased. The parameter update step size is recalculated, and the above steps are continued to obtain a new gain ratio and compare it with the positive threshold until the parameter vector subset is updated.

[0056] In a second aspect, an embodiment of the present invention provides a staged robust camera parameter optimization device, comprising:

[0057] An acquisition unit, configured to acquire and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector;

[0058] An optimization unit, configured to perform a multi-stage nonlinear optimization of a parameter vector, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized sequentially in multiple stages, with the input of a subsequent optimization stage being the output of the previous stage;

[0059] The output unit is used to output the final camera parameters obtained through multiple stages of nonlinear optimization process.

[0060] In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor and a memory. The memory is configured to store one or more computer programs; when the processor executes the one or more computer programs stored in the memory, the electronic device is capable of implementing any of the possible design methods of the first aspect.

[0061] In a fourth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method as described in any one of the above embodiments is implemented.

[0062] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which, when running on an electronic device, enables the electronic device to execute any possible design method of any of the above aspects.

[0063] Beneficial effects: By introducing a phased optimization strategy, all parameters to be estimated (such as intrinsic parameters, distortion coefficients, and extrinsic parameters) are divided into multiple parameter subsets, which are optimized in stages. Each stage uses the results of the previous stage as the initial value, avoiding problems such as excessive solution space and unstable convergence caused by one-time full parameter optimization. This significantly reduces the risk of falling into local optimality and improves the final solution accuracy and convergence success rate.

[0064] The Huber loss function is introduced into each iteration to estimate weights, automatically identifying and suppressing image points with large errors (i.e., outliers). The influence of outliers on parameter updates is reduced by weighting the error vector and Jacobian matrix. This significantly improves the algorithm's stability and anti-interference ability in the presence of abnormal conditions such as image noise, reflections, and inaccurate corner detection.

[0065] A trust region strategy is introduced to dynamically evaluate the effectiveness of the current parameter update by calculating the "gain ratio ρ". The damping factor λ is automatically adjusted based on the gain ratio, precisely controlling the step size and direction to achieve more intelligent and efficient nonlinear optimization. This avoids invalid updates and premature convergence, improving overall optimization speed and resource utilization efficiency.

[0066] All parameters are uniformly organized as one-dimensional vectors, and a parameter mask mechanism is used to flexibly control activation parameters at different stages. The algorithm has a clear flow and a reasonable structure, which facilitates module division, debugging, and function expansion. It is suitable for high-dimensional optimization scenarios such as multi-frame image calibration and complex distortion modeling, and is easy to integrate into the visual software framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Provide a schematic diagram of the method framework of the present invention;

[0068] Figure 2 The present invention provides a schematic diagram of the three-stage optimization process of camera calibration;

[0069] Figure 3 Provides a schematic diagram of the camera calibration optimization iterative sub-process for the present invention;

[0070] Figure 4 This is a structural block diagram of a device provided in one embodiment of the present application;

[0071] Figure 5 This is a structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the specific embodiments of the drawings.

[0073] Example 1

[0074] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the present invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0075] In view of the problems existing in the existing technology, such as Figure 1-2, as shown in the figure, a phased robust camera parameter optimization method is proposed, including:

[0076] Obtain and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector to provide reasonable initial values ​​for the optimization algorithm. The Zhang Zhengyou calibration method is usually used to calculate the camera's:

[0077] Internal parameters (such as focal length fx, fy, principal point coordinates cx, cy);

[0078] Distortion coefficients (such as radial k1, k2, tangential p1, p2, and high-order distortion k3);

[0079] External parameters (rotation and translation vectors of each frame image);

[0080] All the above parameters are integrated into a one-dimensional vector θ (theta) in a unified order to facilitate subsequent unified optimization;

[0081] Performing a multi-stage nonlinear optimization on the parameter vector, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence in multiple stages, with the input of the subsequent optimization stage being the output of the previous stage;

[0082] Using the parameter mask (param_mask) mechanism, the parameters in the θ vector are divided into multiple subsets;

[0083] Activate subsets of parameters in multiple stages, increasing the optimization complexity (from simple parameters to complex distortions):

[0084] Phase 1 (basic model):

[0085] Optimization: internal parameters (fx, fy, cx, cy), basic distortion (k1, k2), all external parameters;

[0086] Freeze: tangential distortion (p1, p2), high-order distortion (k3);

[0087] Stage 2 (introducing tangential distortion):

[0088] Activate: p1, p2, and continue optimizing based on the results of stage 1;

[0089] Stage 3 (introducing high-order distortion):

[0090] Activate: k3, all parameters are involved, and global fine optimization is performed.

[0091] At each stage, the improved Levenberg-Marquardt algorithm is used for nonlinear least squares optimization, combined with Huber loss function weight estimation to weaken outliers;

[0092] The trust region strategy dynamically adjusts the step size to improve efficiency and stability;

[0093] Outputs the final camera parameters obtained after multiple stages of nonlinear optimization process.

[0094] During at least one iteration of the optimization phase, robust weights are calculated and applied to reduce the impact of outliers in the image data on the optimization results. Compared with existing techniques, this phased optimization method simplifies the solution space at each step, significantly improving the algorithm's stability and ensuring the stability and convergence success rate of the final result. Furthermore, by introducing robust weight processing, the interference of noisy data is effectively suppressed, significantly improving the robustness and accuracy of the calibration results. This overall approach balances stability, accuracy, and efficiency, and has significant practical application value.

[0095] In some specific embodiments, the parameter vector includes an intrinsic parameter part, a distortion coefficient part, and an extrinsic parameter part;

[0096] The internal parameter part includes focal length parameters and principal point coordinates.

[0097] The distortion coefficient part includes basic radial distortion coefficient, tangential distortion coefficient and high-order radial distortion coefficient;

[0098] The external parameter part includes a rotation vector and a translation vector.

[0099] Specifically, in the present invention, all parameters to be optimized for the camera are organized into a one-dimensional parameter vector (θ), the structure of which specifically includes the following three parts:

[0100] 1. Intrinsic Parameters

[0101] This is the core parameter that describes the camera imaging model itself and affects the geometric relationship between image points and spatial points.

[0102] Parameter Name Description

[0103] Focal length parameters (fx, fy), which describe the scaling factors of the imaging system in the x and y axis directions, are usually expressed in pixels;

[0104] The principal point coordinates (cx, cy) are the points in the image coordinate system through which the optical axis passes, usually close to the center of the image. They are used to calculate the projection relationship between image coordinates and world coordinates and are a reflection of the intrinsic geometric properties of the camera.

[0105] 2. Distortion Coefficients

[0106] Describes the geometric distortion of images caused by the physical structure of the lens, which usually includes the following three categories:

[0107] The basic radial distortion coefficients k1 and k2 are bending deformations (barrel or pillow), which increase with the increase of radius.

[0108] The tangential distortion coefficients p1 and p2 are asymmetric distortions caused by lens assembly errors, which cause the image to tilt and deform to one side.

[0109] The high-order radial distortion coefficient k3 finely models the stronger distortion effect and is added when high accuracy is required. It is used to dedistort the pixels in the image to make the mapping more consistent with the real projection model.

[0110] 3.Extrinsic Parameters

[0111] Describes the transformation from the world coordinate system to the camera coordinate system, which is used to locate the position and posture of the camera relative to the calibration object.

[0112] The rotation vector (r) represents the rotation of the camera coordinate system relative to the world coordinate system;

[0113] The translation vector (t) represents the translation offset of the camera center relative to the world coordinate system;

[0114] There is a set of external parameters for each frame of image to ensure an accurate correspondence between the three-dimensional points in space and their image projections.

[0115] In some specific embodiments, obtaining and initializing parameters of a camera to be calibrated includes:

[0116] Call Zhang Zhengyou calibration algorithm to obtain the intrinsic parameter part, distortion coefficient part and extrinsic parameter part according to the input three-dimensional point set of the world coordinate system and its corresponding two-dimensional point set of the image coordinate system.

[0117] Specifically, Zhang's Calibration Method is a classic camera calibration method based on a planar calibration plate. It is widely used in industrial vision and computer vision, offering the advantages of computational simplicity and high accuracy. The coordinates of corner points extracted from checkerboard images captured by multiple cameras (a two-dimensional point set in the image coordinate system) and the real-world coordinates of the checkerboard (a three-dimensional point set in the world coordinate system, with the Z coordinate being a constant) are used to estimate the homography matrix for each image. The initial intrinsic parameter matrix is ​​then obtained by jointly solving the matrix, and the extrinsic parameters and distortion coefficients are further estimated by applying orthogonal constraints to the rotation matrix.

[0118] In some specific embodiments, combined Figure 2 , the preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence at different stages, and the input of the subsequent optimization stage is the output result of the previous stage, including:

[0119] The optimization parameters are adjacent in three stages, where:

[0120] In the first stage, a parameter subset is constructed using a parameter mask. This parameter subset includes the focal length parameter and principal point coordinates of the intrinsic parameters, the basic radial distortion coefficients of the distortion coefficients, and the rotation and translation vectors of the extrinsic parameters. The basic model parameters are optimized by setting the mask param_mask to optimize only the core intrinsic parameters fx, fy, cx, and cy, the basic radial distortion parameters k1 and k2, and all extrinsic parameters (rotation and translation for all frames). The tangential distortion p1 and p2 and the high-order radial distortion k3 are frozen (the parameters remain unchanged at this stage) and their values ​​are kept at 0. Levenberg-Marquardt iterative optimization is performed on this parameter subset until convergence conditions are met or the maximum number of iterations is reached.

[0121] In the second phase, a second parameter subset is constructed using a parameter mask. This parameter subset includes the tangential distortion coefficients from the first phase, updates the mask param_mask, introduces the tangential distortion parameters p1 and p2, and freezes the high-order radial distortion parameter k3. Using the optimization results from the first phase as the initial values ​​(including updated parameters (such as intrinsic parameters, k1, k2), as well as those frozen parameters that remain unchanged (for example, p1, p2 remain at 0)), LM iterative optimization is continued on this new, larger parameter subset until convergence or the maximum number of iterations is reached.

[0122] In the third stage, a parameter mask is used to construct a third parameter subset. This parameter subset includes the addition of higher-order radial distortion coefficients that pass through the distortion coefficients in the second stage. The mask param_mask is updated to introduce higher-order distortion and perform global optimization. At this point, all parameters in param_mask are set to true. Using the second stage optimization results as initial values, a global LM iterative optimization is performed on all parameters until convergence or the maximum number of iterations is reached.

[0123] Specifically, the benefits are as follows: The optimization dimension is reduced, convergence efficiency is improved, the parameter dimensions of each stage are low, the solution space is small, convergence is easy and more stable, and the initial introduction of complex distortions that lead to non-convergence or reduced accuracy is avoided. Model complexity is controlled to reduce interference, and the model's degrees of freedom are gradually increased between stages to ensure that the optimization logic progresses from simple to complex; this makes the optimization of each stage more targeted and easier to analyze. Accuracy is significantly improved, and high-order distortion terms are only introduced in the final stage to ensure that they do not interfere with the basic structure. After multiple rounds of iteration, the parameters are closer to the actual physical model, improving the final calibration accuracy.

[0124] In some specific embodiments, combined Figure 2 、 3 The nonlinear optimization process includes optimizing the Levenberg-Marquardt algorithm based on the trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, including:

[0125] The theoretical image point position is calculated based on the current parameter vector, compared with the actual image point to generate an error vector, and the partial derivatives of the error with respect to the activation parameters of the current parameter subset are solved to construct the Jacobian matrix;

[0126] Based on the current parameter estimate, the error between the theoretical projection position and the actual image point is obtained. The current parameter vector θ is used to project the 3D point onto the image point, and the theoretical image point is calculated. The difference between the theoretical image point and the actual image point is then used to form the error vector err. The partial derivative of the error vector with respect to the current "activated parameter subset" is taken to construct the Jacobian matrix J.

[0127] The absolute values ​​of the reprojection errors for all data points are obtained from the error vector. The median absolute deviation is calculated based on the absolute values ​​of the reprojection errors for all data points, and the standard deviation of the error distribution is estimated. Based on this standard deviation, the Huber loss function is used to assign weights to each data point, suppressing the impact of outliers on the overall optimization through a robust mechanism. The absolute values ​​of the reprojection errors for all data points in the error vector are calculated; the median absolute deviation (MAD) is calculated based on these absolute errors, and the standard deviation of the error distribution is further estimated. A weight w_i is assigned to each data point based on the Huber loss function: points with small errors have a weight of 1; points with large errors have a weight less than 1 to reduce their impact.

[0128] The weight is multiplied element-by-element by the error vector to obtain a weighted error vector, and the weight is multiplied row-by-row by the Jacobian matrix to obtain a weighted Jacobian matrix, thereby constructing a robust objective function.

[0129] Perform weighted processing on the error vector err and the Jacobian matrix J: each error term err_i is multiplied by the corresponding weight w_i to obtain err_w; each row of the Jacobian matrix J is multiplied by the weight w_i of the corresponding point to obtain J_w; mathematically, this is equivalent to using the diagonal weight matrix W = diag(w) to process the objective function;

[0130] This is called the Hadamard Product, or element-wise multiplication.

[0131] ;

[0132] Expanded into vector form:

[0133] ;

[0134] If the weight w_i of a point is small (for example, 0.2), then its component err_w_i in the weighted error vector will also be reduced accordingly.

[0135] Weighted Jacobian matrix J_w, for the Jacobian matrix, the weights are applied row by row. Because each row of the Jacobian matrix corresponds to a data point, all elements of this row (that is, the partial derivatives of the error of this point with respect to all parameters) should be multiplied by the weight of this point. Mathematically, this is equivalent to multiplying the original Jacobian matrix with a diagonal matrix on the left Where diag(w) is an NxN diagonal matrix, the elements on its diagonal are the elements of the weight vector w, and the rest of the elements are all 0.

[0136] Expanded into matrix form:

[0137] ;

[0138] If the weight w_i of a point is very small, then all elements of the entire row J_wi corresponding to that point in the Jacobian matrix will be reduced. This means that the gradient contribution of this outlier point to all parameters will be weakened.

[0139] The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size;

[0140] The standard LM equation: , the weighted LM equation can be obtained by replacing all J and err in the above equation with J_w and err_w:

[0141] ;

[0142] In this way, the influence of outliers with small weights w_i in the approximate Hessian matrix on the left (J_w^TJ_w) and the gradient vector on the right (J_w^Terr_w) is greatly reduced, thus achieving the robustness of the algorithm;

[0143] Calculate the gain ratio to evaluate the effectiveness of the update. Based on the gain ratio threshold, decide whether to accept the parameter update and adjust the damping factor. Calculate the gain ratio and compare it with the threshold to decide whether to accept the parameter update. Dynamically adjust the damping factor λ accordingly to achieve iterative optimization of trust region control.

[0144] Convergence means that the parameters have been "almost optimized" and further optimization is meaningless:

[0145] The parameter update step size is very small, which means the update amplitude is small, indicating that the model has become stable.

[0146] The maximum number of iterations (Max Iteration Limit) prevents "falling into an infinite loop" or "convergence is very slow". For example, set the maximum number of iterations to 100; if convergence is not achieved after 100 iterations, it will be forcibly terminated.

[0147] In some specific embodiments, constructing and solving the Levenberg-Marquardt incremental equation based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size includes:

[0148] Substituting the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is as follows:

[0149] ;

[0150] in, is the transpose of the weighted Jacobian matrix, is the weighted Jacobian matrix, is the weighted error vector, is the damping factor, is the identity matrix, is the parameter update step size;

[0151] The approximate Hessian matrix JtJ and gradient g are calculated by the weighted Jacobian matrix J_w and the weighted error vector err_w. The formula is as follows;

[0152] ;

[0153] ;

[0154] By substituting the approximate Hessian matrix JtJ and the gradient g into the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is obtained:

[0155] ;

[0156] Get the parameter update step size, the formula is as follows:

[0157] .

[0158] Specifically, this formula is the LM equation under weighted least squares, which incorporates the trust region concept (controlled by λ) and the practical approximation of the Hessian matrix and gradient terms. The computational efficiency is high and the J constructed using the first-order derivative is T J avoids the calculation of second-order derivatives; realizes simple Jacobian matrix for easy numerical calculation, which is conducive to matrix partitioning and sparse processing; optimizes stability and combines damping factors to form trust region control, improving robustness; and reduces the influence of outliers by more strongly weighting noise, further enhancing robustness.

[0159] In some specific embodiments, calculating the gain ratio to evaluate the update effectiveness, and determining whether to accept the parameter update and adjust the damping factor according to a gain ratio threshold include:

[0160] The candidate parameter vector is calculated as follows:

[0161] ;

[0162] in, is the parameter vector subset of the current input;

[0163] The actual error reduction is calculated as follows:

[0164] ;

[0165] The actual error reduction is calculated as follows:

[0166] ;

[0167] The gain ratio is obtained as follows:

[0168] ;

[0169] Measures the "predictive ability of the model": actual decline / theoretical decline; reflects whether the update is effective and worth accepting. This is the extremely important "trust region" mechanism in the LM algorithm. Although δ is the step size calculated by the LM equation, it is based on a first-order approximation (J^TJ); if the model linear approximation is inaccurate, δ may cause the error to become larger rather than smaller.

[0170] If ρ is large, it means that the model prediction is accurate and the current step direction and size are credible;

[0171] If ρ is very small or even negative, it means that the model prediction fails and this direction (or magnitude) should not be used.

[0172] The gain ratio is compared with the preset positive threshold. If it is greater than the positive threshold, it means that the update is valid, and the parameter vector subset is updated to the candidate parameter vector, while reducing the damping factor;

[0173] If the gain ratio is less than the positive threshold, no update is performed, and the damping factor is increased. The parameter update step size is recalculated, and the above steps are continued to obtain a new gain ratio and compare it with the positive threshold until the parameter vector subset is updated.

[0174] The model prediction is more accurate when the threshold is positive (such as 0.1), and the error does decrease. The update is accepted, the damping λ is reduced, and the step size is increased, which meets the convergence requirements.

[0175] ρ≤positive threshold, model prediction is inaccurate, the error does not decrease significantly or increases, refuse to update, roll back the parameters, increase λ, and reduce the step size;

[0176] By dynamically adjusting the damping factor λ, a gradual transition from "conservative gradient descent" to "bold Gauss-Newton" can be achieved; this is the key to the stability and fast convergence of the LM algorithm.

[0177] Therefore, in each optimization iteration, the effectiveness of the parameter update is evaluated by calculating the ratio between the actual decrease in the sum of squared errors before and after the update and the decrease in the model prediction, i.e., the "gain ratio ρ";

[0178] If ρ exceeds the set threshold, it means that the current step size is valid, then the update is accepted and the damping factor is reduced;

[0179] If ρ is too small or negative, it means that the current update effect is not good, so the update is rejected and the damping factor is increased;

[0180] This mechanism can effectively prevent divergence caused by excessive step size and slow convergence caused by excessive conservatism, thereby achieving a fast, stable and robust nonlinear optimization process.

[0181] In other embodiments of the present invention, Figure 4 , an embodiment of the present invention discloses a staged robust camera parameter optimization device, comprising:

[0182] An acquisition unit 201 is used to acquire and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector;

[0183] An optimization unit 202 is configured to perform a multi-stage nonlinear optimization on the parameter vector, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence in multiple stages, with the input of the subsequent optimization stage being the output of the previous stage;

[0184] The output unit 203 is configured to output the final camera parameters obtained through the multi-stage nonlinear optimization process.

[0185] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0186] In other embodiments of the present invention, an electronic device 400 is disclosed. Figure 5 As shown, the electronic device may include: one or more processors 401; a memory 402; a display 403; one or more applications (not shown); and one or more computer programs 404. The above components may be connected via one or more communication buses 405. The one or more computer programs 404 are stored in the memory 402 and configured to be executed by the one or more processors 401. The one or more computer programs 404 include instructions, which may be used to execute the following: Figures 1 to 2 and each step in the corresponding embodiment.

[0187] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0188] Each functional unit in each embodiment of the present invention may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product, stored on a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) or processor to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, mobile hard disks, read-only memory, random access memory, magnetic disks, or optical disks.

[0190] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A phased robust camera parameter optimization method, characterized in that: include: Obtain and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector; Performing a multi-stage nonlinear optimization on the parameter vector, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence in multiple stages, with the input of the subsequent optimization stage being the output of the previous stage; Outputs the final camera parameters obtained after multiple stages of nonlinear optimization process.

2. The method according to claim 1, characterized in that The parameter vector includes the internal parameter part, the distortion coefficient part and the external parameter part; The internal reference part includes focal length parameters and principal point coordinates; The distortion coefficient part includes basic radial distortion coefficient, tangential distortion coefficient and high-order radial distortion coefficient; The external parameter part includes a rotation vector and a translation vector.

3. The method according to claim 1, characterized in that Get the parameters of the camera to be calibrated and initialize it, including: Call Zhang Zhengyou calibration algorithm to obtain the intrinsic parameter part, distortion coefficient part and extrinsic parameter part according to the input three-dimensional point set of the world coordinate system and its corresponding two-dimensional point set of the image coordinate system.

4. The method according to claim 2, characterized in that The preset parameter mask divides the parameter vector into multiple parameter subsets, and activates and optimizes the multiple parameter subsets in sequence at different stages. The input of the subsequent optimization stage is the output result of the previous stage, including: The optimization parameters are adjacent in three stages, where: In the first stage, a first parameter subset is constructed through a parameter mask, where the first parameter subset includes the focal length parameter and the principal point coordinates of the intrinsic parameter part, the basic radial distortion coefficient of the distortion coefficient part, and the rotation vector and translation vector of the extrinsic parameter part; In the second stage, a second parameter subset is constructed through the parameter mask, wherein the second parameter subset includes a tangential distortion coefficient additionally added through the distortion coefficient part on the basis of the result of the first stage; In the third stage, a third parameter subset is constructed through a parameter mask, wherein the third parameter subset includes a high-order radial distortion coefficient that is additionally added to the distortion coefficient part based on the result of the second stage.

5. The method according to claim 4, characterized in that The nonlinear optimization process includes optimizing the Levenberg-Marquardt algorithm based on the trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, including: The theoretical image point position is calculated based on the current parameter vector, compared with the actual image point to generate an error vector, and the partial derivatives of the error with respect to the activation parameters of the current parameter subset are solved to construct the Jacobian matrix; The absolute value of the reprojection error of all data points is obtained from the error vector, the median absolute deviation is calculated based on the absolute value of the reprojection error of all data points, the standard deviation of the error distribution is estimated, and the Huber loss function is used to assign weights to each data point based on the standard deviation; Multiplying the weight by the error vector element by element to obtain a weighted error vector, and multiplying the weight by the Jacobian matrix row by row to obtain a weighted Jacobian matrix; The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size; The gain ratio is calculated to evaluate the effectiveness of the update, and the gain ratio threshold is used to decide whether to accept the parameter update and adjust the damping factor.

6. The method according to claim 5, characterized in that Based on the weighted Jacobian matrix and the weighted error vector, the Levenberg-Marquardt incremental equation is constructed and solved to obtain the parameter update step size, including: Substituting the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt increment equation, the formula is as follows: ； in, is the transpose of the weighted Jacobian matrix, is the weighted Jacobian matrix, is the weighted error vector, is the damping factor, is the identity matrix, is the parameter update step size; The approximate Hessian matrix JtJ and gradient g are calculated by the weighted Jacobian matrix J_w and the weighted error vector err_w. The formula is as follows; ； ； By substituting the approximate Hessian matrix JtJ and the gradient g into the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt increment equation, the formula is obtained: ； Get the parameter update step size, the formula is as follows: 。 7. The method according to claim 6, characterized in that Calculate the gain ratio to evaluate the effectiveness of the update, decide whether to accept the parameter update and adjust the damping factor based on the gain ratio threshold, including: The candidate parameter vector is calculated as follows: ； in, is the parameter vector subset of the current input; The actual error reduction is calculated as follows: ； The actual error reduction is calculated as follows: ； The gain ratio is obtained as follows: ； The gain ratio is compared with the preset positive threshold. If it is greater than the positive threshold, it means that the update is valid, and the parameter vector subset is updated to the candidate parameter vector, while reducing the damping factor; If the gain ratio is less than the positive threshold, no update is performed, and the damping factor is increased. The parameter update step size is recalculated, and the above steps are continued to obtain a new gain ratio and compare it with the positive threshold until the parameter vector subset is updated.

8. A staged robust camera parameter optimization device, characterized in that: include: An acquisition unit, configured to acquire and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector; An optimization unit, configured to perform a multi-stage nonlinear optimization of a parameter vector, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized sequentially in multiple stages, with the input of a subsequent optimization stage being the output of the previous stage; The output unit is used to output the final camera parameters obtained through multiple stages of nonlinear optimization process.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images