Staged robust camera parameter optimization method, apparatus, device and medium

Abstract

The application discloses a kind of phased robust camera parameter optimization method, device, equipment and medium, it is related to camera parameter optimization field, comprising: obtaining the parameter of camera to be calibrated and initialization, and the parameter is organized in unified parameter vector;Parameter vector is carried out multiple stages of nonlinear optimization, wherein, preset parameter mask divides parameter vector into multiple parameter subsets, and multiple parameter subsets are sequentially activated optimization in multiple stages, and the input of the latter optimization stage is the output result of the former stage;Output final camera parameter obtained by multiple stages of nonlinear optimization process.Main technical scheme and effect: by introducing phased optimization strategy, all to be estimated parameters are divided into multiple parameter subsets, and are sequentially optimized in stages;The result of the former stage is used as the initial value of each stage, to avoid the problems such as too large solution space and unstable convergence caused by one-time full parameter optimization;Significantly reduce the risk of falling into local optimum.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of camera parameter optimization, and in particular to a staged robust camera parameter optimization method, device, equipment and medium. BACKGROUND

[0002] With the rapid development of machine vision and computer vision technology, camera calibration as an indispensable key step in the vision system, its accuracy directly affects the performance of subsequent image processing and three-dimensional reconstruction tasks. At present, the widely used camera calibration method is mainly based on the plane calibration method proposed by Zhang Zhengyou, which uses a simple plane checkerboard as a calibration board, and the camera takes pictures from multiple angles, and the intrinsic and extrinsic parameters of the camera are obtained by solving the homography matrix between the image and the world coordinates.

[0003] Although the Zhang Zhengyou method greatly simplifies the calibration process, and the accuracy can be further improved by Levenberg-Marquardt (LM) nonlinear optimization algorithm, but the optimization process usually puts all the parameters to be estimated (including intrinsic, extrinsic, and multiple order distortion coefficients) into the same optimization framework at one time. Due to the high dimension of the parameter, the solution space is complex, and this kind of overall optimization method is easy to fall into local optimum, unstable convergence, and even not convergent, which affects the stability and accuracy of the calibration.

[0004] In addition, in actual sampling, due to uncontrollable factors such as image noise, corner extraction error, and reflection interference, there are inevitably outliers with large errors in the image point data. The traditional LM algorithm takes minimizing the square error as the goal, and it is difficult to suppress the influence of these outliers on the optimization result, so that the final calibration result deviates from the true value, and the robustness is poor. At the same time, the adjustment strategy of the existing LM algorithm for damping factor is rough, usually only relies on fixed iteration number as the termination condition, cannot adaptively judge whether it is convergent, and affects the overall optimization efficiency.

[0005] In summary, the following problems still exist in the prior art:

[0006] Optimizing all parameters at one time leads to high dimension of optimization space, easy to fall into local optimum, poor convergence;

[0007] Lack of effective suppression mechanism for image outliers, resulting in low robustness of optimization results;

[0008] The parameter updating strategy lacks dynamic adjustment ability, and the optimization efficiency is not high. SUMMARY

[0009] The purpose of the present application is to solve the technical problems in the prior art, and to provide a staged robust camera parameter optimization method, device, equipment and medium.

[0010] The one or more embodiments of the specification also relate to a staged robust camera parameter optimization device, an electronic device, a computer readable storage medium and a computer program product to solve the technical defects in the prior art.

[0011] Technical solutions:

[0012] In a first aspect, the present application provides a staged robust camera parameter optimization method, comprising:

[0013] obtaining and initializing the parameters of the camera to be calibrated, and organizing the parameters in a unified parameter vector;

[0014] performing multiple stages of nonlinear optimization on the parameter vector, wherein the parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in multiple stages in sequence, and the input of the next optimization stage is the output result of the previous stage;

[0015] outputting the final camera parameters obtained through the multiple stages of nonlinear optimization.

[0016] Preferably, the parameter vector includes an intrinsic part, a distortion coefficient part and an extrinsic part.

[0017] The intrinsic part includes focal length parameters and principal point coordinates.

[0018] The distortion coefficient part includes a basic radial distortion coefficient, a tangential distortion coefficient and a high-order radial distortion coefficient.

[0019] The extrinsic part includes a rotation vector and a translation vector.

[0020] Preferably, obtaining and initializing the parameters of the camera to be calibrated includes:

[0021] calling Zhang Zhengyou's calibration algorithm to obtain the intrinsic part, the distortion coefficient part and the extrinsic part according to the input three-dimensional point set of the world coordinate system and the corresponding two-dimensional point set of the image coordinate system.

[0022] Preferably, the parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in different stages in sequence, and the input of the next optimization stage is the output result of the previous stage, comprising:

[0023] optimizing the parameters through three stages, wherein:

[0024] The first stage constructs a first parameter subset through a parameter mask, wherein the first parameter subset includes the focal length parameters and the principal point coordinates of the intrinsic part, the basic radial distortion coefficient of the distortion coefficient part, and the rotation vector and the translation vector of the extrinsic part.

[0025] The second stage constructs a second parameter subset by parameter mask, wherein the second parameter subset includes additional tangential distortion coefficients through the distortion coefficient part based on the first stage result;

[0026] The third stage constructs a third parameter subset by parameter mask, wherein the third parameter subset includes additional high-order radial distortion coefficients through the distortion coefficient part based on the second stage result.

[0027] Preferably, the nonlinear optimization process includes Levenberg-Marquardt algorithm optimization based on trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, wherein it includes:

[0028] Calculate the theoretical image point position based on the current parameter vector, generate the error vector by comparing with the real image point, and solve the partial derivative of the error to the active parameters of the current parameter subset to construct the Jacobian matrix;

[0029] Get the absolute value of the re-projection error of all data points from the error vector, calculate the median absolute deviation according to the absolute value of the re-projection error of all data points, estimate the standard deviation of the error distribution, and assign weights to each data point based on the standard deviation using the Huber loss function;

[0030] Multiply the weights and the error vector element by element to obtain the weighted error vector, and multiply the weights and the Jacobian matrix row by row to obtain the weighted Jacobian matrix;

[0031] Construct and solve the Levenberg-Marquardt increment equation based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step;

[0032] Calculate the gain ratio to evaluate the effectiveness of the update, and decide whether to accept the parameter update and adjust the damping factor according to the positive threshold of the preset gain ratio.

[0033] Preferably, the Levenberg-Marquardt increment equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step, including:

[0034] Substitute the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt increment equation, and the formula is as follows:

[0035] ;

[0036] Wherein, is the transpose of the weighted Jacobian matrix, is the weighted Jacobian matrix, is the weighted error vector, is the damping factor, is the unit matrix, is the parameter update step;

[0037] The approximate Hessian matrix JtJ and the gradient g are calculated by the weighted Jacobian matrix J_w and the weighted error vector err_w, and the formula is as follows:

[0038]

[0039]

[0040] The formula is obtained by substituting the approximate Hessian matrix JtJ and the gradient g into the standard Levenberg-Marquardt increment equation:

[0041]

[0042] The parameter update step is obtained, and the formula is as follows:

[0043]

[0044] Preferably, the gain ratio is calculated to evaluate the effectiveness of the update, and according to a positive threshold of a preset gain ratio, it is determined whether to accept the parameter update and adjust the damping factor, including:

[0045] The candidate parameter vector is calculated, and the formula is as follows:

[0046]

[0047] wherein, is the current input parameter vector subset;

[0048] The actual error reduction amount is calculated, and the formula is as follows:

[0049]

[0050] The predicted error reduction amount is calculated, and the formula is as follows:

[0051]

[0052] The gain ratio is obtained, and the formula is as follows:

[0053]

[0054] If the gain ratio is greater than the positive threshold, it indicates that the update is effective, and the parameter vector subset is updated to the candidate parameter vector, and the damping factor is reduced.

[0055] If the gain ratio is less than the positive threshold, no update is performed, and the damping factor is increased, the parameter update step is recalculated, and the above steps are continued to obtain a new gain ratio compared with the positive threshold, until the parameter vector subset is updated. ​​​​​​​​

[0056] In a second aspect, an embodiment of the present application provides a staged robust camera parameter optimization device, comprising:

[0057] An acquisition unit is configured to acquire parameters of a camera to be calibrated and initialize, and organize the parameters in a unified parameter vector;

[0058] An optimization unit is configured to perform nonlinear optimization on the parameter vector in multiple stages, wherein a preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in sequence in multiple stages, and an input of a later optimization stage is an output result of a previous stage;

[0059] An output unit is configured to output final camera parameters obtained through the nonlinear optimization in multiple stages.

[0060] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory. The memory is configured to store one or more computer programs. When the one or more computer programs stored in the memory are executed by the processor, the electronic device is enabled to implement the method of any possible design of the first aspect.

[0061] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

[0062] In a fifth aspect, an embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the method of any possible design of any of the above aspects.

[0063] Advantages: By introducing a staged optimization strategy, all to-be-estimated parameters (such as intrinsic parameters, distortion coefficients, and extrinsic parameters) are divided into multiple parameter subsets and are optimized in sequence in stages. The result of a previous stage is used as an initial value for each stage, thereby avoiding problems such as too large solution space and unstable convergence caused by one-time full-parameter optimization. The risk of falling into local optimum is significantly reduced, and the final solution accuracy and convergence success rate are improved.

[0064] In each iteration, a Huber loss function is introduced to estimate weights, to automatically identify and suppress image points with large errors (i.e., outliers). By performing weighted processing on an error vector and a Jacobian matrix, the influence of outliers on parameter updating is weakened. The stability and anti-interference ability of the algorithm in the presence of image noise, glare, inaccurate corner detection, and other abnormal situations are significantly improved.

[0065] The reliability domain strategy is introduced, the current parameter update is dynamically evaluated whether it is effective by calculating "gain ratio p", the gain ratio is automatically adjusted according to the damping factor, the step and direction are accurately controlled, the more intelligent and efficient nonlinear optimization is realized, invalid update and premature convergence are avoided, and the overall optimization speed and resource utilization efficiency are improved;

[0066] All parameters are uniformly organized as a one-dimensional vector, the parameter mask mechanism is used to flexibly control the activated parameters in different stages, the algorithm flow is clear, the structure is reasonable, module division, debugging and function expansion are facilitated, and the method is suitable for multi-frame image calibration, complex distortion modeling and other high-dimensional optimization scenes, and is convenient for integration in a visual software framework. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 A method framework schematic diagram is provided for the application;

[0068] Figure 2 A camera calibration three-stage optimization flow schematic diagram is provided for the application;

[0069] Figure 3 A camera calibration optimization iteration sub-flow schematic diagram is provided for the application;

[0070] Figure 4 It is a device structure block diagram provided by an embodiment of the application;

[0071] Figure 5 It is an electronic equipment structure block diagram provided by an embodiment of the application. DETAILED DESCRIPTION

[0072] In order to make the technical solutions of the application clearer, the following will make a further detailed description of the application in combination with specific embodiments and the accompanying drawings.

[0073] Embodiment 1

[0074] In order to make the purpose, technical solutions and advantages of the application clearer, the following will make a clear and complete description of the technical solutions in the embodiments of the application in combination with the drawings of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application. Unless otherwise defined, the technical terms or scientific terms used herein should be understood as the usual meanings understood by those skilled in the art in the field of the application. The "includes" and similar words used herein mean that the elements or objects before the words cover the elements or objects listed after the words and their equivalents, without excluding other elements or objects.

[0075] In view of the problems in the prior art, such as Figures 1-2, as shown, a phased robust camera parameter optimization method is proposed, comprising:

[0076] The parameters of the camera to be calibrated are obtained and initialized, and the parameters are organized in a unified parameter vector to provide reasonable initial values for the optimization algorithm. Zhang Zhengyou calibration method is usually used. According to the corresponding relationship between the two-dimensional corner points in the captured image and the three-dimensional world coordinate points, the camera's:

[0077] Intrinsic parameters (such as focal length fx, fy, principal point coordinates cx, cy);

[0078] Distortion coefficients (such as radial k1, k2, tangential p1, p2, high-order distortion k3);

[0079] Extrinsic parameters (rotation and translation vectors of each frame of image);

[0080] Integrate all the above parameters into a one-dimensional vector θ (theta) in a unified order for subsequent unified optimization;

[0081] The parameter vector is optimized in multiple stages, wherein the parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in multiple stages, and the input of the next optimization stage is the output result of the previous stage;

[0082] Using the parameter mask (param_mask) mechanism, the parameters in the θ vector are divided into multiple subsets;

[0083] According to the optimization complexity from low to high, the parameter subsets are gradually activated in multiple stages (from simple parameters to complex distortion):

[0084] Stage one (basic model):

[0085] Optimization: intrinsic parameters (fx, fy, cx, cy), basic distortion (k1, k2), all extrinsic parameters;

[0086] Frozen: tangential distortion (p1, p2), high-order distortion (k3);

[0087] Stage two (introducing tangential distortion):

[0088] Activate: p1, p2, and continue optimization based on the stage one result;

[0089] Stage three (introducing high-order distortion):

[0090] Activate: k3, all parameters participate, and perform global fine optimization.

[0091] In each stage, a modified Levenberg-Marquardt algorithm is used for nonlinear least squares optimization, combined with: Huber loss function weight estimation, weakening outliers;

[0092] The confidence domain strategy dynamically adjusts the step size, improving efficiency and stability.

[0093] The final camera parameters obtained through the nonlinear optimization process of multiple stages are output.

[0094] During the iteration process of at least one optimization stage, robust weights are calculated and applied to reduce the influence of outliers in image data on the optimization results. Compared with the prior art, the present application simplifies the solution space of each step through staged optimization, significantly improves the stability of the algorithm to ensure the stability and convergence success rate of the final results; at the same time, by introducing robust weight processing, the interference of noise data is effectively suppressed, and the robustness and accuracy of the calibration results are greatly improved. The overall method takes into account stability, accuracy and efficiency, and has important practical application value.

[0095] In some specific embodiments, the parameter vector includes an intrinsic part, a distortion coefficient part, and an extrinsic part.

[0096] The intrinsic part includes focal length parameters and principal point coordinates.

[0097] The distortion coefficient part includes a basic radial distortion coefficient, a tangential distortion coefficient, and a high-order radial distortion coefficient.

[0098] The extrinsic part includes a rotation vector and a translation vector.

[0099] Specifically, in the present application, all the to-be-optimized parameters of the camera are uniformly organized into a one-dimensional parameter vector (θ), which specifically includes the following three parts:

[0100] 1. Intrinsic Parameters

[0101] This is the core parameter that describes the camera imaging model itself, which affects the geometric relationship between image points and space points.

[0102] Parameter name Description

[0103] Focal length parameters (fx, fy), which describe the scaling factors of the imaging system in the x and y axis directions, usually in pixels;

[0104] Principal point coordinates (cx, cy), which are the points through which the optical axis passes in the image coordinate system, usually close to the image center; used to calculate the projection relationship between image coordinates and world coordinates, and is a manifestation of the camera's intrinsic geometric properties.

[0105] 2. Distortion Coefficients

[0106] The image geometric distortion caused by the physical structure of the lens is usually described as follows:

[0107] The basic radial distortion coefficients k1, k2 bend deformation (barrel or pincushion), which increases with the increase of the radius;

[0108] The tangential distortion coefficients p1, p2 are caused by the lens assembly error, which makes the image tilt deformation to one side;

[0109] The high-order radial distortion coefficient k3 is used to model the stronger distortion effect, which is added when the accuracy requirement is high; it is used to correct the distortion of the pixel points in the image, so that the mapping is more consistent with the real projection model.

[0110] 3. Extrinsic Parameters

[0111] The transformation from the world coordinate system to the camera coordinate system is described, which is used to locate the position and attitude of the camera relative to the calibration object.

[0112] The rotation vector (r) represents the rotation of the camera coordinate system relative to the world coordinate system;

[0113] The translation vector (t) represents the translation offset of the camera center relative to the world coordinate system;

[0114] There is a set of extrinsic parameters for each frame of image, which ensures the accurate correspondence between the spatial three-dimensional points and their image projections.

[0115] In some specific embodiments, the parameters of the camera to be calibrated are obtained and initialized, including:

[0116] The Zhang calibration algorithm is called to obtain the intrinsic parameters, distortion coefficients and extrinsic parameters according to the input three-dimensional point set in the world coordinate system and the corresponding two-dimensional point set in the image coordinate system.

[0117] Specifically, Zhang’s Calibration Method is a classic camera calibration method based on a planar calibration board, widely used in industrial vision and computer vision fields, with the advantages of simple calculation and high precision. The corner point coordinates (two-dimensional point set in the image coordinate system) extracted from the checkerboard images taken by multiple cameras, and the plane coordinates of the checkerboard in the real world (three-dimensional point set in the world coordinate system, Z coordinate is constant), estimate the homography matrix of each image, and solve the initial intrinsic matrix by simultaneous equations. Further estimate the extrinsic parameters and distortion coefficients by orthogonal constraint of rotation matrix.

[0118] In some specific embodiments, combinedFigure 2 The preset parameter mask divides the parameter vector into multiple parameter subsets, and sequentially activates and optimizes the multiple parameter subsets in different stages, and the input of a later optimization stage is the output result of a previous stage, comprising:

[0119] The optimization parameter is adjacent through three stages, wherein:

[0120] The first stage constructs a first parameter subset through a parameter mask, wherein the first parameter subset includes focal length parameters and principal point coordinates of an intrinsic part, basic radial distortion coefficients of a distortion coefficient part, and rotation vectors and translation vectors of an extrinsic part, optimizes basic model parameters, sets a mask param_mask, only optimizes core intrinsic parameters fx, fy, cx, cy, basic radial distortion parameters k1, k2, and all extrinsic parameters (rotation and translation of all frames), freezes (the parameters are fixed and unchanged in this stage) tangential distortion p1, p2 and high-order radial distortion k3, and the values remain 0. On the parameter subset, Levenberg-Marquardt (Levenberg-Marquardt) iterative optimization is performed until the convergence condition is met or the maximum number of iterations is reached.

[0121] The second stage constructs a second parameter subset through a parameter mask, wherein the second parameter subset includes additional tangential distortion coefficients through the distortion coefficient part on the basis of the first stage result, updates the mask param_mask, introduces the tangential distortion parameters p1, p2, and freezes the high-order radial distortion parameter k3. The optimization result of the first stage is used as the initial value (including the updated parameters (such as intrinsic parameters, k1, k2), and also including the parameters that are maintained as originally (for example, p1, p2 remains 0) because they are frozen), and the LM iterative optimization is continued on the new and larger parameter subset until convergence or the maximum number of iterations is reached.

[0122] The third stage constructs a third parameter subset through a parameter mask, wherein the third parameter subset includes additional high-order radial distortion coefficients through the distortion coefficient part on the basis of the second stage result, updates the mask param_mask, introduces the high-order distortion, and performs global optimization. At this point, all parameters in param_mask are true, the optimization result of the second stage is used as the initial value, and the LM iterative optimization is performed on all parameters until convergence or the maximum number of iterations is reached.

[0123] Specifically, there are the following beneficial effects: reducing the optimization dimension, improving the convergence efficiency, the parameter dimension of each stage optimization is low, the solution space is small, easy to converge and more stable, avoiding not converging or precision decline caused by introducing complex distortion at the beginning. Control model complexity, reduce interference, gradually increase model freedom between stages, ensure that the optimization logic is simple to complex; make the optimization of each stage more targeted and easier to analyze. The precision is improved significantly, the high-order distortion term is introduced only in the last stage, which ensures not to interfere with the basic structure; after multiple iterations, the parameters are closer to the real physical model, improving the final calibration accuracy.

[0124] In some specific embodiments, in combination Figure 2 、 3 , the nonlinear optimization process includes optimization by Levenberg-Marquardt algorithm based on trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, wherein it includes:

[0125] Based on the current parameter vector, the theoretical image point position is calculated, the error vector is generated by comparing with the real image point, and the partial derivative of the error to the active parameter of the current parameter subset is solved to construct the Jacobian matrix;

[0126] Based on the current parameter estimation, the error between the theoretical projection position and the real image point is obtained. The projection of a three-dimensional point to an image point is calculated using the current parameter vector θ, and the theoretical image point is calculated. The difference between the real image point and the error vector err is formed. The partial derivative of the error vector to the current "active parameter subset" is calculated, and the Jacobian matrix J is constructed.

[0127] The absolute value of the re-projection error of all data points is obtained from the error vector, the median absolute deviation is calculated based on the absolute value of the re-projection error of all data points, the standard deviation of the error distribution is estimated, and the Huber loss function is used to assign weights to each data point based on the standard deviation. The influence of outliers on the overall optimization is suppressed through a robust mechanism. The absolute value of the re-projection error of all data points in the error vector is calculated; based on these absolute errors, the median absolute deviation (MAD) is calculated, and the standard deviation of the error distribution is further estimated; based on the Huber loss function, each data point is assigned a weight w_i: the weight of a small error point is 1; the weight of a large error point is less than 1, reducing its influence;

[0128] The weight and the error vector are multiplied element by element to obtain a weighted error vector, and the weight and the Jacobian matrix are multiplied row by row to obtain a weighted Jacobian matrix, and a robust objective function is constructed.

[0129] The error vector err and the Jacobian matrix J are weighted: each error term err_i is multiplied by the corresponding weight w_i to get err_w; each row of the Jacobian matrix J is multiplied by the weight w_i of the corresponding point to get J_w; mathematically, this is equivalent to processing the objective function with a diagonal weight matrix W = diag(w);

[0130] This is called the Hadamard Product, or element-wise multiplication.

[0131] ;

[0132] In vector form, this is:

[0133] ;

[0134] If the weight w_i of a point is small (e.g. 0.2), then its component err_w_i in the weighted error vector will also be correspondingly small.

[0135] The weighted Jacobian matrix J_w, for the Jacobian matrix J, the weights are applied row-wise. Since each row of the Jacobian matrix corresponds to a data point, the entire row of elements (i.e. the partial derivatives of the error for this one point with respect to all parameters) should be multiplied by the weight of this one point. Mathematically, this is equivalent to left-multiplying the original Jacobian matrix by a diagonal matrix where diag(w) is an NxN diagonal matrix whose diagonal elements are the elements of the weight vector w, and all other elements are zero.

[0136] In matrix form, this is:

[0137] ;

[0138] If the weight w_i of a point is small, then all elements of the entire row J_w_i of the Jacobian matrix corresponding to this point will be reduced. This means that the gradient contribution of this outlier point with respect to all parameters will be weakened.

[0139] Based on the weighted Jacobian matrix and the weighted error vector, the Levenberg-Marquardt increment equation is constructed and solved to obtain the parameter update step;

[0140] The standard LM equation: The weighted LM equation replaces all J and err in the above equation with J_w and err_w:

[0141] ;

[0142] In this way, the influence of outliers with very small weights w_i on the approximate Hessian matrix (J_w^TJ_w) on the left and the gradient vector (J_w^Terr_w) on the right is greatly reduced, thus achieving the robustness of the algorithm.

[0143] The gain ratio is calculated to evaluate the effectiveness of the update. Based on the preset positive threshold of the gain ratio, it is decided whether to accept the parameter update and adjust the damping factor. The gain ratio is calculated and compared with the threshold to decide whether to accept the parameter update. The damping factor λ is dynamically adjusted accordingly to achieve iterative optimization of trust region control.

[0144] Convergence indicates that the parameters have been "optimized sufficiently," and further optimization is not very meaningful.

[0145] A small parameter update step size indicates a small update magnitude, suggesting that the model has stabilized.

[0146] The maximum number of iterations (Max Iteration Limit) is used to prevent "getting stuck in an infinite loop" or "slow convergence". For example, setting the maximum number of iterations to 100 will force termination if convergence is not achieved after 100 iterations.

[0147] In some specific embodiments, the Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size, including:

[0148] By substituting the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is as follows:

[0149] ;

[0150] in, For the transpose of the weighted Jacobian matrix, For a weighted Jacobian matrix, This is the weighted error vector. The damping factor, It is the identity matrix. Update the step size for the parameters;

[0151] The approximate Hessian matrix JtJ and gradient g are calculated using the weighted Jacobian matrix J_w and the weighted error vector err_w, as shown in the following formula;

[0152] ;

[0153] ;

[0154] By substituting the approximate Hessian matrix JtJ and gradient g into the weighted Jacobian matrix and weighted error vector, and then into the standard Levenberg-Marquardt incremental equation, we obtain the following formula:

[0155] ;

[0156] The parameter update step size is obtained using the following formula:

[0157] .

[0158] Specifically, this formula is the LM equation under weighted least squares, incorporating the concept of trust regions (controlled by λ), and practically approximating the Hessian matrix and gradient terms. It is computationally efficient, using the first derivative to construct the J... T J avoids calculating the second derivative; it achieves simple Jacobian matrix numerical computation, which is beneficial for matrix partitioning and sparse processing; it optimizes stability by combining damping factor to form trust region control, thereby improving robustness; and it further enhances robustness by reducing the impact of outliers after stronger noise weighting.

[0159] In some specific embodiments, the gain ratio is calculated to evaluate the effectiveness of the update, and a decision is made on whether to accept the parameter update and adjust the damping factor based on a preset positive threshold for the gain ratio, including:

[0160] The candidate parameter vector is calculated using the following formula:

[0161] ;

[0162] in, This is a subset of the currently input parameter vector;

[0163] The reduction in actual error is calculated using the following formula:

[0164] ;

[0165] The reduction in prediction error is calculated using the following formula:

[0166] ;

[0167] The gain ratio is obtained using the following formula:

[0168] ;

[0169] Measuring the "predictive power of the model": actual decrease / theoretical decrease; reflects whether the update is effective and acceptable. This is the extremely important "trust region" mechanism in the LM algorithm. Although δ is the step size calculated through the LM equation, it is based on the first-order approximation (J^TJ). If the linear approximation of the model is inaccurate, δ may cause the error to increase rather than decrease.

[0170] If ρ is large, it means that the model prediction is accurate and the current step direction and size are reliable;

[0171] If ρ is very small or even negative, it indicates that the model prediction has failed, and this direction (or magnitude) should not be used.

[0172] If the gain ratio is greater than the preset positive threshold, the update is considered valid. The parameter vector subset is then updated to a candidate parameter vector while the damping factor is reduced.

[0173] If the gain ratio is less than the positive threshold, no update is performed. Instead, the damping factor is increased, the parameter update step size is recalculated, and the above steps are repeated to obtain a new gain ratio and compare it with the positive threshold until the parameter vector subset is updated.

[0174] With a positive threshold (e.g., 0.1), the model prediction is more accurate, the error does decrease, and an update is accepted. The damping λ is reduced, the step size is increased, and the convergence requirement is met.

[0175] If ρ ≤ positive threshold, the model prediction is inaccurate, the error has not decreased significantly or has increased, refuse to update, roll back the parameters, increase λ, and decrease the step size;

[0176] By dynamically adjusting the damping factor λ, a gradual transition from "conservative gradient descent" to "bold Gauss-Newton" can be achieved; this is the key to the stability and fast convergence of the LM algorithm.

[0177] Therefore, in each optimization iteration, the effectiveness of the parameter update is evaluated by calculating the ratio between the actual decrease in the sum of squared errors before and after the update and the model's predicted decrease, i.e., the "gain ratio ρ".

[0178] If ρ exceeds the set threshold, it means that the current step size is valid, so the update is accepted and the damping factor is reduced;

[0179] If ρ is too small or negative, it indicates that the current update effect is not good, so the update is rejected and the damping factor is increased;

[0180] This mechanism can effectively prevent divergence caused by excessively large step sizes, and also avoid slow convergence due to excessive conservatism, thereby achieving a fast, stable, and robust nonlinear optimization process.

[0181] In other embodiments of the invention, combined with Figure 4 This invention discloses a staged robust camera parameter optimization device, comprising:

[0182] The acquisition unit 201 is used to acquire and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector;

[0183] The optimization unit 202 is used to perform nonlinear optimization on the parameter vector in multiple stages. The preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in multiple stages in sequence. The input of the later optimization stage is the output result of the previous stage.

[0184] Output unit 203 is used to output the final camera parameters obtained after a multi-stage nonlinear optimization process.

[0185] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0186] In other embodiments of the present invention, an electronic device 400 is disclosed, such as... Figure 5 As shown, the electronic device may include: one or more processors 401; a memory 402; a display 403; one or more application programs (not shown); and one or more computer programs 404. These devices can be connected via one or more communication buses 405. The one or more computer programs 404 are stored in the memory 402 and configured to be executed by the one or more processors 401. The one or more computer programs 404 include instructions that can be used to perform actions such as... Figures 1 to 2 And the steps in the corresponding embodiments.

[0187] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0188] In the various embodiments of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0190] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present invention should be covered within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. A staged robust camera parameter optimization method, characterized in that, include: Obtain and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector; The parameter vector is subjected to nonlinear optimization in multiple stages. A preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in multiple stages in sequence. The input of the later optimization stage is the output of the previous stage. Output the final camera parameters obtained through a multi-stage nonlinear optimization process; The nonlinear optimization process includes optimization using the Levenberg-Marquardt algorithm based on the trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, including: The theoretical image point position is calculated based on the current parameter vector, compared with the real image point to generate an error vector, and the partial derivative of the error with respect to the activation parameters of the current parameter subset is solved to construct the Jacobian matrix. The absolute value of the reprojection error of all data points is obtained from the error vector. The median absolute deviation is calculated based on the absolute value of the reprojection error of all data points. The standard deviation of the error distribution is estimated. Based on the standard deviation, the Huber loss function is used to assign weights to each data point. The weights are multiplied element by element by the error vector to obtain the weighted error vector, and the weights are multiplied row by row by the Jacobian matrix to obtain the weighted Jacobian matrix; The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size. The calculation of the gain ratio evaluates the effectiveness of the update, and decides whether to accept the parameter update and adjust the damping factor based on the preset positive threshold of the gain ratio; The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and weighted error vector to obtain the parameter update step size, including: By substituting the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is as follows: ； in, For the transpose of the weighted Jacobian matrix, For a weighted Jacobian matrix, This is the weighted error vector. The damping factor, It is the identity matrix. Update the step size for the parameters; The approximate Hessian matrix JtJ and gradient g are calculated using the weighted Jacobian matrix J_w and the weighted error vector err_w, as shown in the following formula; ； ； By substituting the approximate Hessian matrix JtJ and gradient g into the standard Levenburg-Marquardt incremental equation, we obtain the following formula: ； The parameter update step size is obtained using the following formula: ； Whether to accept parameter updates and adjust the damping factor is determined based on a preset positive threshold for the gain ratio, including: The candidate parameter vector is calculated using the following formula: ； in, This is a subset of the currently input parameter vector; The reduction in actual error is calculated using the following formula: ； in, This represents a subset of the updated error parameter vector. This is a subset of the error parameter vector before the update. The reduction in prediction error is calculated using the following formula: ； The gain ratio is obtained using the following formula: ； If the gain ratio is greater than the preset positive threshold, the update is considered valid. The parameter vector subset is then updated to a candidate parameter vector while the damping factor is reduced. If the gain ratio is less than the positive threshold, no update is performed. Instead, the damping factor is increased, the parameter update step size is recalculated, and the above steps are repeated to obtain a new gain ratio and compare it with the positive threshold until the parameter vector subset is updated.

2. The method according to claim 1, characterized in that, The parameter vector includes intrinsic parameters, distortion coefficients, and extrinsic parameters; The intrinsic parameters include focal length parameters and principal point coordinates; The distortion coefficients include the basic radial distortion coefficient, the tangential distortion coefficient, and the higher-order radial distortion coefficients; The extrinsic parameters include rotation vectors and translation vectors.

3. The method according to claim 2, characterized in that, Obtain and initialize the parameters of the camera to be calibrated, including: The Zhang Zhengyou calibration algorithm is invoked to obtain the intrinsic parameters, distortion coefficients, and extrinsic parameters based on the input three-dimensional point set in the world coordinate system and its corresponding two-dimensional point set in the image coordinate system.

4. The method according to claim 2, characterized in that, A preset parameter mask divides the parameter vector into multiple parameter subsets, and these subsets are sequentially activated and optimized at different stages. The input to each subsequent optimization stage is the output of the previous stage, including: The parameter vector is optimized in three stages, where: The first stage constructs the first parameter subset through a parameter mask, which includes the focal length parameter and principal point coordinates of the intrinsic parameter part, the basic radial distortion coefficient of the distortion coefficient part, and the rotation vector and translation vector of the extrinsic parameter part. The second stage constructs a second parameter subset through a parameter mask, which includes the tangential distortion coefficients passed through the distortion coefficient part on the basis of the result of the first stage. The third stage constructs a third parameter subset through a parameter mask. This third parameter subset includes higher-order radial distortion coefficients obtained by adding the distortion coefficients to the results of the second stage.

5. A staged robust camera parameter optimization device, characterized in that, include: The acquisition unit is used to acquire and initialize the parameters of the camera to be calibrated, and organize the parameters into a unified parameter vector; The optimization unit is used to perform nonlinear optimization of the parameter vector in multiple stages. The preset parameter mask divides the parameter vector into multiple parameter subsets, and the multiple parameter subsets are activated and optimized in multiple stages in sequence. The input of the later optimization stage is the output result of the previous stage. The output unit is used to output the final camera parameters obtained after a multi-stage nonlinear optimization process. The nonlinear optimization process includes optimization using the Levenberg-Marquardt algorithm based on the trust region strategy at each stage until the convergence condition is met or the maximum number of iterations is reached, including: The theoretical image point position is calculated based on the current parameter vector, compared with the real image point to generate an error vector, and the partial derivative of the error with respect to the activation parameters of the current parameter subset is solved to construct the Jacobian matrix. The absolute value of the reprojection error of all data points is obtained from the error vector. The median absolute deviation is calculated based on the absolute value of the reprojection error of all data points. The standard deviation of the error distribution is estimated. Based on the standard deviation, the Huber loss function is used to assign weights to each data point. The weights are multiplied element by element by the error vector to obtain the weighted error vector, and the weights are multiplied row by row by the Jacobian matrix to obtain the weighted Jacobian matrix; The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and the weighted error vector to obtain the parameter update step size. The calculation of the gain ratio evaluates the effectiveness of the update, and decides whether to accept the parameter update and adjust the damping factor based on the preset positive threshold of the gain ratio; The Levenberg-Marquardt incremental equation is constructed and solved based on the weighted Jacobian matrix and weighted error vector to obtain the parameter update step size, including: By substituting the weighted Jacobian matrix and the weighted error vector into the standard Levenberg-Marquardt incremental equation, the formula is as follows: ； in, For the transpose of the weighted Jacobian matrix, For a weighted Jacobian matrix, This is the weighted error vector. The damping factor, It is the identity matrix. Update the step size for the parameters; The approximate Hessian matrix JtJ and gradient g are calculated using the weighted Jacobian matrix J_w and the weighted error vector err_w, as shown in the following formula; ； ； By substituting the approximate Hessian matrix JtJ and gradient g into the standard Levenburg-Marquardt incremental equation, we obtain the following formula: ； The parameter update step size is obtained using the following formula: ； Whether to accept parameter updates and adjust the damping factor is determined based on a preset positive threshold for the gain ratio, including: The candidate parameter vector is calculated using the following formula: ； in, This is a subset of the currently input parameter vector; The reduction in actual error is calculated using the following formula: ； in, This represents a subset of the updated error parameter vector. This is a subset of the error parameter vector before the update. The reduction in prediction error is calculated using the following formula: ； The gain ratio is obtained using the following formula: ； If the gain ratio is greater than the preset positive threshold, the update is considered valid. The parameter vector subset is then updated to a candidate parameter vector while the damping factor is reduced. If the gain ratio is less than the positive threshold, no update is performed. Instead, the damping factor is increased, the parameter update step size is recalculated, and the above steps are repeated to obtain a new gain ratio and compare it with the positive threshold until the parameter vector subset is updated.

6. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, causes the processor to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program therein, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images