Zoom camera focal length error correction method based on image processing

Abstract

This invention belongs to the field of image processing technology and relates to a method for correcting focal length errors in zoom cameras based on image processing. The method includes: acquiring multi-level calibration board images of the zoom camera; calculating the neighborhood gradient direction entropy pixel-by-pixel based on the RGB three channels to generate an entropy-weighted multi-channel gradient field fusion image; constructing a Gaussian scale space pyramid; obtaining the initial coordinates and feature scales of candidate calibration points through multi-scale Harris-Laplace response and scale space non-maximum suppression; building a local gray-level topological model related to illumination affine distortion and scale geometric constraints; solving for the sub-pixel coordinates of the calibration points and calculating the location confidence; iteratively optimizing the calibration point coordinates and camera intrinsic parameters; and establishing a mapping model between the nominal focal length and the actual focal length after convergence to complete the focal length error correction. This invention can accurately correct the focal length error of zoom cameras and improve the accuracy of calibration and vision tasks.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for correcting focal length error of zoom cameras based on image processing. Background Technology

[0002] In the field of computer vision, tasks such as high-precision 3D reconstruction, target tracking, and dimensional measurement place stringent requirements on the accuracy of camera parameters. Zoom cameras, due to their ability to flexibly switch between different fields of view and resolutions, are widely used in these complex scenarios. However, due to the gaps in the mechanical structure and the complexity of the optical system design, the internal parameters of a zoom camera change dynamically and non-linearly with the zoom level. In this physical process, there is an unavoidable objective error between the nominal focal length indicated on the device or the theoretical focal length value directly output by the control system and the actual focal length corresponding to the actual image formed after the light passes through the complex lens group. If this focal length error is not accurately corrected, it will directly lead to a significant decrease in the accuracy of all subsequent visual tasks that rely on geometric projection relationships, or even the output of completely incorrect measurement and reconstruction results.

[0003] To address the aforementioned parameter deviation issues, the industry commonly employs camera calibration techniques based on planar calibration targets. This technique typically requires capturing numerous images of a planar calibration board containing specific patterns using a camera at multiple preset discrete zoom levels. Subsequently, corner detection or center detection algorithms are used to extract the pixel coordinates of these calibration points in the two-dimensional images, and these coordinates are matched with the known three-dimensional coordinates of the calibration board in the physical world. Based on these spatially matched point pairs, traditional solutions often employ mature algorithms such as the Zhang Zhengyou calibration method to solve for the camera's internal parameters at each discrete zoom level. Finally, a mathematical mapping relationship between the nominal focal length and the actual focal length is established through polynomial and other function fitting methods, thereby correcting the error.

[0004] While existing technologies have alleviated the focal length error problem to some extent, they still face numerous technical bottlenecks in practical applications. First, in the front-end calibration point feature extraction stage, traditional detection algorithms are extremely sensitive to external environmental factors. When faced with complex lighting changes, image blur, severe perspective shrinkage, and lens distortion, the stability of the algorithms significantly decreases. Moreover, these algorithms are typically forced to operate on a single grayscale image channel, losing color information and resulting in low positioning accuracy of the calibration points. Second, in the sub-pixel level precise positioning stage, existing local window grayscale models cannot effectively cope with affine distortion caused by tilted shooting angles and uneven lighting, nor do they incorporate the changes in the projected size of the same calibration point at different zoom scales, thus limiting the final accuracy of coordinate extraction. Furthermore, in the global parameter optimization stage, traditional methods typically treat all detected calibration points equally, making the optimization results susceptible to the influence of a few low-quality points; simultaneously, they fail to fully utilize the inherent mesh structure prior of the calibration board to constrain the optimization process, thereby failing to improve the stability and global consistency of the solution. Summary of the Invention

[0005] The purpose of this invention is to propose a zoom camera focal length error correction method based on image processing, in order to solve the problems of low positioning accuracy of calibration points in existing zoom camera calibration methods when dealing with complex lighting and perspective distortion, and the lack of effective geometric constraints in the global optimization process, which leads to inaccurate focal length error correction results and poor stability.

[0006] Therefore, the present invention provides a zoom camera focal length error correction method based on image processing, comprising: Acquire calibration board images captured by a zoom camera at multiple different zoom levels; for any calibration board image, calculate the gradient direction entropy in the neighborhood of each pixel based on the R, G, and B channels, and generate a multi-channel gradient field fusion image with the reciprocal of the gradient direction entropy as the weight. A Gaussian scale space pyramid is constructed on the multi-channel gradient field fused image. The initial coordinates and feature scales of the candidate calibration points are obtained by calculating the multi-scale Harris-Laplace response and performing non-maximum suppression in the joint domain of scale space. For each candidate calibration point, a two-dimensional grayscale template representing the ideal corner pattern with a fixed size is constructed, and a corresponding reference scale is defined. On the image layer corresponding to the feature scale, a local neighborhood grayscale topological model is established, relating local illumination affine distortion and geometric projection size constraints related to the feature scale. This model expresses the predicted grayscale values ​​of the image within the local neighborhood of the candidate calibration point as a function of the illumination affine parameters, the affine transformation matrix, the two-dimensional grayscale template, and the sub-pixel coordinate offset of the calibration point. The expression for this function is: ,in, Predict grayscale values ​​for the corresponding image. For illumination contrast parameters, This is the brightness offset parameter. and The affine parameters constituting the illumination are This represents the ideal grayscale value of the two-dimensional grayscale template at the corresponding coordinate position. For the horizontal template coordinate variable, The vertical template coordinates are variables; the actual pixel coordinates of the image and the coordinates of the two-dimensional grayscale template satisfy the coordinate relationship transformation equation. , and The initial coordinates of the candidate calibration points. and These are the sub-pixel coordinate offsets in the horizontal and vertical directions, respectively. It is the affine transformation matrix; and Let x and y be the x and y coordinates of any actual pixel in the local neighborhood of the candidate calibration point, respectively. Based on the feature scale of the candidate calibration point, the initial value of the determinant of the affine transformation matrix is ​​constrained to the square of the ratio of the feature scale to the reference scale. The residual between the image predicted gray value calculated by the local neighborhood gray-scale topology model and the image actual gray value is minimized through an iterative algorithm. The sub-pixel coordinate offset is solved to update the sub-pixel coordinates of the candidate calibration point, and the location confidence, which is inversely proportional to the residual, is calculated. The sub-pixel coordinates of all calibration points in all calibration board images and the camera parameters at each zoom level are alternately and iteratively optimized until convergence. After iterative convergence, the nominal focal length and the optimized actual focal length at each zoom level are extracted, and a mapping relationship model between the two is established to complete the focal length error correction.

[0007] This invention achieves stable extraction and accurate initial localization of calibration points through multi-channel gradient field fusion weighted by RGB three-channel gradient direction entropy and multi-scale Harris-Laplace feature detection; it achieves high-precision localization and quantification of positional confidence of calibration points through a local grayscale topology model that correlates illumination and geometric constraints; it achieves globally consistent calculation of camera internal parameters through joint optimization iteration; and it ultimately solves the problem of decreased accuracy and measurement errors in visual tasks caused by multi-level focal length errors of zoom cameras by mapping modeling between nominal focal length and actual focal length.

[0008] Preferably, the method for acquiring the multi-channel gradient field fusion image includes: For R, G, and B three-channel images, a 3×3 Sobel operator is used to calculate the gradient magnitude and gradient direction of each pixel in each channel. For each pixel, the gradient direction is represented as eight directional intervals in a 9×9 spatial neighborhood, and the gradient magnitude is used as the weight to calculate the gradient direction histogram corresponding to each channel. The gradient direction entropy of each pixel corresponding to each channel is calculated based on the gradient direction histogram, and the reciprocal of the gradient direction entropy is extracted as a weight. The gradient magnitudes of the three channels are weighted and averaged according to the weights corresponding to each channel to obtain a multi-channel gradient field fusion image.

[0009] This invention avoids feature loss caused by single-channel imaging defects. At the same time, by weighting the gradient direction entropy, it highlights the effective gradient information in the edge and corner regions of the image, suppresses invalid noise interference in smooth regions, and further improves the quality of gradient field fusion images.

[0010] Preferably, obtaining the initial coordinates and feature scale of the candidate calibration points includes: The Gaussian scale spatial pyramid is constructed using a set initial scale and scale factor. The Gaussian scale spatial pyramid contains multiple sets, and each set contains multiple scale layers. At each scale level, a 2×2 autocorrelation matrix is ​​calculated, and the Harris corner response value is calculated. The Harris corner response value is the determinant of the autocorrelation matrix minus the product of an empirical constant and the square of the trace of the autocorrelation matrix. In the scale dimension, the extreme values ​​of the Laplacian Gaussian response of each pixel are calculated to determine the feature scale; Within the scale-space joint domain, non-maximum suppression is performed on the Harris corner response in a 3×3×3 neighborhood. The suppressed maxima are determined as candidate calibration points, the two-dimensional coordinates of the maxima are used as the initial coordinates, and the scale value of the scale layer is used as the feature scale.

[0011] This invention constructs a Gaussian scale spatial pyramid, which can cover calibration point features at different scales and adapt to the imaging scale changes of zoom cameras at different settings. Through multi-scale corner response calculation and non-maximum suppression of the scale space joint domain, it can accurately screen out stable corner features and eliminate interference from false corners and noise points.

[0012] Preferably, the calculation of the location reliability, which is inversely proportional to the residual, includes: The minimum squared difference of the output after the iterative algorithm converges is obtained as the numerical residual; Divide the constant 1 by the sum of the numerical residual and the correction parameter to prevent the denominator from being zero, and obtain the initial confidence level. The initial confidence scores of all candidate calibration points within a single calibration board image are uniformly normalized, and the normalized values ​​are determined as the location confidence scores.

[0013] This invention quantifies the physical reliability of the positioning coordinates of each calibration point, while the normalization process eliminates the basic residual scale deviation caused by the overall illumination difference between different images.

[0014] Preferably, the alternating iterative optimization process performs parameter updates and coordinate updates.

[0015] Preferably, the parameters are updated to fix the current sub-pixel coordinates of all candidate calibration points, and the camera parameters at each zoom level are updated based on the camera calibration model. The camera parameters include focal length, principal point, and distortion coefficient. The coordinates are updated to fix the updated camera parameters, and the sub-pixel coordinates of all candidate calibration points are globally consistent by minimizing a joint energy function. The joint energy function is the sum of a weighted projection error term and a regularization term. The calculation process of the weighted projection error term includes: using the currently updated camera parameters, projecting the theoretical position of the calibration board in the three-dimensional model coordinate system onto the two-dimensional image plane to obtain the coordinates of the theoretical projection point; calculating the norm square of the deviation distance between the theoretical projection point coordinates of each candidate calibration point and the currently updated sub-pixel coordinates to obtain the reprojection error; multiplying the reprojection error of each candidate calibration point by the corresponding position confidence and performing a numerical weighted summation to obtain the weighted projection error term. The calculation process of the regularization term includes: for any three calibration points on the calibration board image that are on the same physical straight line, calculate the geometric distance from the middle projection coordinate to the straight line determined by the other two projection coordinates, and sum the squares of all such geometric distances as the collinearity constraint energy term; for multiple sets of parallel straight lines on the calibration board image, establish corresponding intersection points as vanishing points in the homogeneous coordinate system, and calculate the sum of the squares of the inner products of the homogeneous algebraic vectors of each set of straight lines and the homogeneous coordinate vectors of the corresponding vanishing points as the vanishing point constraint energy term; and sum the collinearity constraint energy term and the vanishing point constraint energy term according to a preset weighting coefficient to form the regularization term.

[0016] This invention further improves the global rationality of calibration point coordinates, ensuring that the camera parameter calculation results conform to the geometric laws of actual imaging.

[0017] Preferably, the step of extracting the nominal focal length and the optimized actual focal length at each zoom level, establishing a mapping relationship model between the two, and completing the focal length error correction includes: Extract the pairing data of nominal focal length and actual focal length for all zoom levels; A polynomial model is set to fit the mapping relationship between the nominal focal length and the actual focal length; The least squares method is used to find the polynomial coefficients that minimize the sum of squared fitting errors for all paired data. The solved polynomial coefficients are solidified and used to correct any input nominal focal length value to the corresponding actual focal length value.

[0018] The beneficial effects of this invention are as follows: By utilizing multi-channel gradient directional entropy fusion to enhance the noise resistance of calibration point features and constructing a local model relating illumination distortion and geometric constraints, sub-pixel localization of calibration points under complex illumination and geometric deformation is achieved. In the global optimization stage, by using the confidence level calculated from the localization residual to weight the reprojection error and fusing prior mesh structures with collinearity and vanishing point constraints, the interference of outliers and local localization errors on the overall solution process is suppressed, ensuring the global geometric consistency of the calibration point coordinates and achieving calibration and correction of the focal length error of the zoom camera across the entire zoom range. Attached Figure Description

[0019] Figure 1 This is a flowchart of the zoom camera focal length error correction method based on image processing in this embodiment. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0021] like Figure 1 As shown, the zoom camera focal length error correction method based on image processing proposed in this embodiment includes the following steps: S1. Acquire calibration board images taken by the zoom camera at multiple different zoom levels; for any calibration board image, calculate the gradient direction entropy in the neighborhood of each pixel based on the R, G, and B channels, and generate a multi-channel gradient field fusion image with the reciprocal of the gradient direction entropy as the weight.

[0022] Specifically, the zoom camera is controlled to adjust the zoom level step by step from the wide-angle end to the telephoto end. At each set zoom level, multiple calibration board images containing checkerboard patterns or dot array patterns are captured from different spatial perspectives and shooting distances, and the nominal focal length value reported by the internal firmware of the zoom camera at the corresponding time is recorded simultaneously.

[0023] For the single-channel grayscale image separated from the R, G, and B channels, a 3×3 Sobel edge detection operator is used to calculate the gradient magnitude of each pixel. With direction Set the channel identifier variable to ,in These represent the red, green, and blue color channels, respectively. When applying the standard Sobel edge detection operator, the horizontal kernel matrix is ​​set... for Vertical kernel matrix for For each specific channel any pixel position The gradient magnitude is calculated through the convolution operation of the kernel matrix mentioned above. and the corresponding gradient direction gradient magnitude and the corresponding gradient direction They respectively satisfy the following formulas:

[0024]

[0025] In the above formula, and These represent the results of the convolution of the partial derivatives of the pixels in the horizontal and vertical directions, respectively. It is a bivariate arctangent function used to accurately calculate the gradient direction angle covering the entire circumference.

[0026] After obtaining the gradient magnitude and gradient direction values ​​of a single pixel, a 9×9 pixel spatial neighborhood is defined centered on that pixel. Within this spatial neighborhood, continuous gradient direction values ​​are linearly discretized and divided into 8 preset number of direction intervals. A gradient direction histogram containing 8 statistical intervals is constructed for the central pixel based on this division standard. Then, all pixels within the 9×9 spatial neighborhood are traversed, and the corresponding gradient magnitude is accumulated into the specific interval into which the gradient direction of each neighboring pixel falls. After completing the traversal and accumulation operation of this neighborhood, the values ​​of the gradient direction histogram are normalized to obtain probability distribution data reflecting the local texture structure characteristics of the pixel. , where the parameters This represents the traversal sequence number of the interval from the 1st to the 8th direction.

[0027] Based on the extracted probability distribution data, the reciprocal of the Shannon information entropy is calculated as the dynamic weight for subsequent fusion operations. The specific weight extraction formula is as follows:

[0028] In this formula, Represents the center pixel In the specified color channel The gradient direction entropy weights are calculated above. The summation symbol indicates the summation of the logarithmic product terms related to the probability values ​​of the eight direction intervals. This is a base-2 logarithmic mathematical operation. Parameters as well as These are both system-preset minimal positive real constants. The purpose of introducing these two constants is to provide numerical stability, prevent illegal logarithm calculations when the probability value is 0, and avoid overflow of weight values ​​caused by the entire denominator being 0.

[0029] After calculating the entropy weights of the three independent color channels, the gradient magnitude image data corresponding to the red, green, and blue channels are extracted respectively, i.e., the corresponding parameters are extracted. , as well as The gradient magnitude of each pixel in the three channels is calculated based on the entropy weights of each channel at the same location. , as well as A pixel-by-pixel weighted averaging operation is performed to generate the final multi-channel gradient field fused image. The specific calculation formula is as follows:

[0030] In this formula, The multi-channel gradient field fused image representing the final output is located at coordinates. The overall gradient pixel intensity value at that location. An additional value appended to the end of the denominator of the formula. Similarly, this is a very small positive number used to prevent division-by-zero anomalies. The fused image calculated by this method can automatically assign the maximum weight to the color channel with the clearest edge structure and the least noise, thus providing a high-quality image feature foundation for subsequent scale space construction and sub-pixel precise localization.

[0031] S2. Construct a Gaussian scale space pyramid on the multi-channel gradient field fusion image, and obtain the initial coordinates and feature scale of the candidate calibration points by calculating the multi-scale Harris-Laplace response and performing non-maximum suppression in the joint domain of scale space.

[0032] This step uses a predetermined initial scale and scale factor to construct a Gaussian scale space pyramid by continuously performing Gaussian smoothing filtering and image downsampling operations. This Gaussian scale space pyramid consists of multiple pyramid groups, each containing multiple scale layers. The initial scale variable is set as follows: The scaling factor variable is Then, in this Gaussian-scale spatial pyramid, the first... The scale parameters of each scale layer are expressed as follows: ,parameter It represents the ascending sequence number of the current scale layer within its group.

[0033] For each specific scale layer in the completed Gaussian-scale spatial pyramid Calculate the horizontal gradient component obtained by convolving the image of this layer with a Gaussian first-order derivative kernel. and vertical gradient components For each pixel at the current scale layer, a Gaussian weighted window is used to weight and statistically analyze the two gradient components mentioned above within the local spatial neighborhood of that pixel, thereby constructing a 2×2 autocorrelation matrix. In this embodiment, the autocorrelation matrix The summation symbol represents a weighted summation operation within a Gaussian weighted window.

[0034] After obtaining the autocorrelation matrix for each pixel, the Harris corner response value, used to determine local texture features, is further calculated. The Harris corner response value is the product of the determinant of the autocorrelation matrix and the square of the trace of the autocorrelation matrix. The formula for calculating this corner response value is as follows: In the formula, the parameter This represents the final Harris corner response value. Represents the autocorrelation matrix The calculated matrix determinant value, Represents the autocorrelation matrix The sum of the elements on the main diagonal is the trace of the matrix, and the parameter is... This represents an empirical constant used to adjust sensitivity. To ensure detection performance, the numerical range of this empirical constant is preferably set to 0.04~0.06.

[0035] While determining the corner features in the two-dimensional spatial domain, it is necessary to calculate the Laplace Gaussian response value of each pixel in the scale dimension to determine its feature scale. For each pixel to be evaluated in the multi-scale space, its normalized Laplace Gaussian response value at different scale levels is calculated, and the calculation formula is as follows: In the formula, The parameter represents the calculated Laplace response value of Gauss. This represents the scale parameter value of the scale layer where the current pixel is located. The parameter represents the second-order partial derivative of the current scale layer image calculated in the horizontal direction. This represents the second-order partial derivative of the current scale layer image calculated in the vertical direction. For absolute value calculation, the Laplacian Gaussian response values ​​of pixels with the same spatial coordinates are compared layer by layer at adjacent scale levels. When a local maximum is detected at a certain level, the scale value of the scale level corresponding to the maximum value is established as the feature scale of the pixel.

[0036] To accurately extract stable and reliable feature points from massive pixels, non-maximum suppression is applied to all calculated Harris corner response values ​​within a three-dimensional scale space joint domain, which includes a two-dimensional spatial dimension and a one-dimensional scale dimension. The specific screening mechanism is as follows: For each target pixel, a three-dimensional joint domain containing a total of 27 adjacent spatial pixel neighborhoods is constructed, including the 3×3 neighboring spatial pixel neighborhoods at the current scale layer and the corresponding 3×3 spatial pixel neighborhoods at the adjacent previous and next scale layers. All Harris corner response values ​​are compared within this three-dimensional joint domain. If the Harris corner response value of the current pixel is a maximum value within this local range, and this maximum value exceeds the system's preset response threshold, and the Gaussian Laplace response value of that point also reaches an extreme value in the scale dimension, then the maximum value point retained after the above non-maximum suppression step is formally determined as a candidate calibration point. Finally, the pixel coordinates of the two-dimensional image plane where the maximum point is located are extracted as the initial coordinates for the subsequent optimization process, and the scale value corresponding to the scale layer where the maximum point is located is recorded as the feature scale of the candidate calibration point.

[0037] S3. For each candidate calibration point, establish a local neighborhood gray-scale topology model on the image layer corresponding to the feature scale, which is associated with local illumination affine distortion and geometric projection size constraints related to the feature scale. By minimizing the residual between the local neighborhood gray-scale topology model and the actual gray value, the sub-pixel coordinates of the calibration point are obtained, and the location confidence, which is inversely proportional to the residual, is calculated.

[0038] Specifically, a local computational region is determined based on the feature scale of the candidate calibration points. This local computational region is the neighborhood window, and its side length is set to be directly proportional to the feature scale. The specific proportionality coefficient is set to a preset constant. Next, a two-dimensional grayscale template representing the ideal corner pattern with a fixed size is constructed. For example, a black and white checkerboard pattern with a size of 15×15 pixels is used as this two-dimensional grayscale template. The horizontal template coordinate variable in the coordinate system of this two-dimensional grayscale template is defined as... And the vertical template coordinate variable is , using parameters This represents the ideal grayscale value of the two-dimensional grayscale template at the corresponding coordinate position, and simultaneously defines a reference feature scale corresponding to the two-dimensional grayscale template, and sets the reference scale variable. The value is 1.

[0039] The initial coordinates of the currently extracted candidate calibration points are represented as the x-coordinates. and ordinate The extracted feature scale is represented as When establishing a local neighborhood grayscale topology model, the x-coordinate of any actual pixel within the local neighborhood of the candidate calibration point is... and ordinate The actual grayscale value of the image at that location is represented as And represent the predicted grayscale value of the image corresponding to that position as The predicted grayscale value of the image is modeled using a linear mapping relationship as a function of illumination contrast parameters, illumination offset parameters, a two-dimensional affine transformation matrix, a two-dimensional grayscale template, and the sub-pixel coordinate offset of the calibration point. In this expression, the parameter Represents the illumination contrast parameter, parameter These two parameters together constitute the illumination affine parameter, which is used to compensate for uneven illumination caused by shooting in the real physical environment.

[0040] In addition to mapping grayscale values, the actual pixel coordinates of the image and the coordinates of the two-dimensional grayscale template also need to be constrained and associated through spatial geometric transformations. The specific coordinate association transformation equation is as follows:

[0041] The equation means that the column vector formed by the actual pixel coordinate deviations is equal to the affine transformation matrix in two-dimensional space. Multiply by the column vector formed by the coordinates of the 2D grayscale template. The first row of the column vector formed by the deviation of the actual pixel coordinates contains the x-coordinates of the actual pixels. Subtract the initial x-coordinate offset from horizontal sub-pixel coordinates The difference between the sums, with the second row containing the y-coordinates of the actual pixels. Subtract the initial ordinate Offset of sub-pixel coordinates in the vertical direction The difference between the sums. Affine transformation matrix in two-dimensional space. It is a two-row, two-column parameter matrix containing four elements. The first row of the column vector formed by the two-dimensional grayscale template coordinates represents the horizontal template coordinate variables. The second row of elements represents the vertical template coordinate variables. In this correlation transformation equation, the horizontal sub-pixel coordinate offset Offset of sub-pixel coordinates in the vertical direction Together, they constitute the sub-pixel coordinate offset of the calibration point to be solved.

[0042] To address the geometric deformation caused by perspective effects and ensure the convergence stability of the nonlinear optimization solution, this step introduces a geometric prior constraint. This constraint is based on the previously determined characteristic scale. The aforementioned two-dimensional affine transformation matrix The initial value of the determinant is set to the square of the ratio of the feature scale to the reference scale, expressed as: In this constraint formula, the left side of the equation... The value of the determinant formed by the calculated affine transformation matrices in two-dimensional space is represented by the right-hand side of the equation, which indicates the characteristic scale. With reference scale The square of the ratio of the two parameters. This equation constraint, from the perspective of physical space transformation, ensures that the macroscopic scale of the sub-pixel localization model has accurate initial prior knowledge before optimization begins.

[0043] Combining the constructed local neighborhood gray-level topology model and geometric association model, an objective function is established for iterative solution. The objective function minimizes the sum of squared differences between the predicted gray-level values ​​calculated by the local neighborhood gray-level topology model and the actual gray-level values ​​of the image using an iterative algorithm. In this embodiment, the mathematical expression of the objective function is: In the formula, The symbol represents the operation of finding the minimum value of the subsequent summation expression. The summation symbol represents the summation operation of the squared gray-level differences of all pixels in the local neighborhood of the current candidate calibration point. After constructing the objective function, a numerical optimization algorithm based on the nonlinear least squares principle is used, such as the Levenberg-Marquardt iterative optimization algorithm or the Gauss-Newton optimization algorithm. Through multiple rounds of residual feedback iterative approximation, the specific values ​​of a total of 8 core parameters that make the above objective function reach its minimum value are jointly solved. These 8 core parameters specifically include the two affine illumination parameters defined above, namely the illumination contrast parameter. and illumination offset parameters The aforementioned two-dimensional affine transformation matrix The four internal spatial matrix elements, and the sub-pixel coordinate offsets of the calibration points in the horizontal and vertical directions. and .

[0044] After the numerical iterative optimization algorithm reaches convergence, the final calculation results are extracted and a coordinate update operation is performed. The initial coordinates of the current candidate calibration points are added to the solved sub-pixel coordinate offsets of the calibration points in the corresponding directions, thus updating the horizontal coordinates to the initial horizontal coordinates. With horizontal offset The numerical values ​​are summed, and the vertical axis is updated to the initial ordinate. With vertical offset The system sums the numerical values ​​and uses this updated value pair as the sub-pixel coordinates of the candidate calibration point in the final output. Simultaneously, the system records the minimum sum of squared residuals output when the iterative calculation converges as the numerical residual, and calculates the location confidence of the current candidate calibration point based on this numerical residual. Specifically, the location confidence is calculated by dividing a constant 1 by the sum of the aforementioned numerical residual and a pre-set minimum correction parameter to prevent the denominator from being zero. This division result yields the initial confidence of the candidate calibration point. The introduction of the minimum positive constant effectively prevents overflow errors caused by dividing by zero. Subsequently, the initial confidence of all candidate calibration points within a single calibration board image is uniformly normalized, and the normalized value is established as the location confidence, thus providing an accurate confidence weight evaluation basis for subsequent global consistency optimization of all calibration board images. The specific normalization operation is as follows: divide the initial confidence of the candidate calibration point by the sum of the initial confidence of all candidate calibration points contained in the same calibration plate image to obtain the localization confidence of the candidate calibration point.

[0045] S4 performs iterative optimization of the sub-pixel coordinates of all calibration points in all calibration board images and the camera parameters at each zoom level until convergence.

[0046] The iterative optimization process alternately performs parameter updates and coordinate updates. This process controls loop termination by setting a strict convergence criterion, such as whether the mathematical norm of the changes in all parameters between two consecutive iterations is less than a preset constant threshold, or whether the total change in the system's reprojection error is less than the preset constant threshold, which is set to 1e-6. In each iteration, the two independent steps of parameter updates and coordinate updates are performed alternately.

[0047] The operational principle of the parameter update step is to keep the current sub-pixel coordinates of all candidate calibration points unchanged, treat these sub-pixel coordinates as known two-dimensional image observation points, and combine them with the corresponding three-dimensional entity coordinates pre-determined in the three-dimensional model coordinate system of the calibration board to construct multiple sets of mapping point pairs from the two-dimensional image plane to the three-dimensional physical space. Based on the extracted mapping point pair sets, the Zhang Zhengyou camera calibration model is used to solve and update the camera parameters at each zoom level. The camera parameters that need to be updated specifically include an intrinsic parameter matrix characterizing the internal perspective projection relationship. This intrinsic parameter matrix further includes horizontal and vertical focal length parameters, horizontal principal point coordinate parameters, and vertical principal point coordinate parameters. At the same time, the camera parameters also include radial and tangential distortion coefficients used to correct the physical optical path deformation of the lens.

[0048] After the parameter update is completed, the coordinate update step is executed immediately. The operation principle of the coordinate update step is to keep the currently updated camera parameters unchanged, and to perform global consistency optimization on the sub-pixel coordinates of all candidate calibration points by minimizing a joint energy function that includes a weighted projection error term and a geometric prior regularization term.

[0049] Specifically, for the first The first calibration plate image The candidate calibration points, whose current two-dimensional sub-pixel coordinates are obtained after solving the sub-pixel localization model in the previous step, are represented as follows: The coordinates of this point in the physical space are represented as follows: Using the latest estimated camera parameters from the current iteration, specifically the intrinsic parameter matrix corresponding to the zoom level. And the extrinsic rotation and translation matrix reflecting the camera's spatial relative pose when the image frame was captured. , coordinates of the 3D model Projecting the image onto the current 2D image sensor plane yields the theoretical projection point, whose coordinates are: Among them, the function This represents the continuous algebraic operation process of transforming a standard 3D rigid body into a 2D spatial perspective projection. After obtaining the theoretical projection point coordinates, the norm squared of the deviation distance between the theoretical projection point coordinates and the currently updated sub-pixel coordinates of each candidate calibration point is calculated to obtain the reprojection error. The reprojection error of each candidate calibration point is multiplied by the corresponding positional confidence and numerically weighted and summed to construct a weighted projection error term. Its mathematical formula is as follows: In the formula, the multiplicand term represents the currently updated subpixel coordinates. coordinates of the theoretical projection point The norm square of the Euclidean deviation distance between them, multiplier term This represents the location confidence level assigned individually to this specific candidate calibration point. The location confidence level is calculated according to step S3.

[0050] The regularization term incorporates prior knowledge of mesh structure based on collinearity constraints and vanishing point geometry, aiming to provide realistic physical geometric constraints for unconstrained point cloud migrations. First, a collinearity constraint energy term is constructed. For any three calibration points that lie on the same physical straight line on the calibration plate image. , as well as Their pixel projection coordinates onto the current image's two-dimensional plane are respectively represented as: , as well as Within the homogeneous coordinate system framework of algebraic projective geometry, passing through the projected coordinates... and The straight line vector between two points It can be obtained through the spatial cross product of these two homogeneous coordinate point vectors, i.e. Subsequently, the intermediate projected coordinates, which were originally physically collinear but have been offset by noise, are calculated. Deviation from the straight line vector The square of the geometric projection distance is calculated using the following formula: In the formula, the numerator represents the square of the algebraic distance obtained by transposing the extracted straight line vector and performing an algebraic inner product with the column vector of point coordinates. The denominator represents the independent sum of squares of the first two components of the straight line vector, i.e., the elements of the straight line normal vector. The collinearity constraint energy term for constraining point cloud smoothness is obtained by globally summing the squares of these geometric distances calculated individually for all point sets within the same verification image that should conform to the collinear geometric distribution rule. .

[0051] Secondly, construct the vanishing point constraint energy term. For a set of multiple parallel straight lines on the calibration plate image, the set of projected straight lines obtained after perspective transformation and imaging by the projection camera is: These lines intersect at the same vanishing point with specific coordinates. Within the same two-dimensional homogeneous coordinate system of mathematics, by constructing an overdetermined system of linear equations for the extracted parallel lines and solving for its least-squares optimal solution using the singular value decomposition algorithm, the homogeneous coordinate vector of the intersection point, i.e., the vanishing point, corresponding to this specific set of parallel lines can be accurately established. Calculate the homogeneous algebraic vector of each set of lines. Homogeneous coordinate vector of the corresponding vanishing point The sum of squares of the inner product serves as the vanishing point constraint energy term, and its specific calculation formula is as follows: The summation symbol indicates that the vector inner product square operation is performed on each extracted line within the set of parallel lines, followed by an accumulation statistical algorithm. This accumulation result constitutes the vanishing point constraint energy term for perspective parallelism consistency in the constraint space. Finally, the collinearity constraint energy term and the vanishing point constraint energy term are summed according to a preset weighting coefficient to form the complete regularization term, whose mathematical expression is: The introduced preset weighting coefficient and Its core function is to balance the relative control weight of the prior geometric distance penalty and the aforementioned pure coordinate position weighted projection error term in the entire numerical nonlinear optimization equation network.

[0052] The fully calculated regularization term is directly mathematically added to the previously obtained weighted projection error term to obtain the complete joint energy function describing the global overall deviation. Then, using an existing nonlinear numerical solution toolkit based on graph structure theory optimization, the final constructed joint energy function is solved by gradient-based minimization descent iteration. This completes the global consistency space update operation of the sub-pixel coordinates of all candidate calibration points under the protection of prior knowledge of the physical structure.

[0053] S5. After iterative convergence, extract the nominal focal length and the optimized actual focal length for each zoom level, establish a mapping relationship model between the two, and complete the focal length error correction.

[0054] After global optimization convergence, for each zoom level Extract nominal focal length recording data from the camera hardware control firmware report file or the underlying raw metadata stream of the image strongly associated with the image captured at this shooting speed. and the corresponding actual physical focal length data obtained through calibration and optimization. The actual physical focal length is the arithmetic mean of the sum of the horizontal and vertical focal length parameters separated from the matrix. By continuously performing the above paired data extraction operation on all zoom levels that are traversed and scanned, a reliable set of numerically paired data can be collected and obtained. .

[0055] To accurately describe the nonlinear mechanical and optical deviations between these two focal length physical quantities throughout the entire continuous zoom mechanical travel, a specific highest power order is defined as... A continuous polynomial algebraic model of order 1 is used to smoothly fit the mathematical mapping relationship between the nominal focal length and the actual focal length. The expression of this basic polynomial algebraic model is: η is the index variable for summing polynomial powers. These are the coefficients of the corresponding terms. This step uses the power order. Taking the conventional quadratic polynomial fitting model equal to 2 as an example, this fitting model can be concretized and directly expressed as: Employing the least-squares curve fitting optimization method from classical numerical approximation analysis theory, a large-scale overdetermined linear equation system is constructed in memory space based on all the aforementioned numerical pairing data records that have been extracted without loss. In the completed linear algebraic system equations, matrix variables... This represents a specific Vandermonde matrix structure consisting of nominal focal length values ​​extracted from each zoom level, arranged in ascending order according to a set polynomial power. (Column vector variables) This represents the column vector of unknown polynomial weight coefficients that the system urgently needs to solve. Column vector variables Represents the column vector of rigid observation target constraints It is composed of the actual focal length values ​​calculated for each zoom level, arranged in sequence.

[0056] By directly solving the normal equation within the domain of fundamental matrix algebra that minimizes the sum of squared fitting errors for all numerically paired datasets, the globally optimal polynomial coefficient solution is ultimately obtained. The optimal system polynomial coefficients obtained from the matrix operation solution of this normal equation are the sequence parameters. By accurately substituting all the polynomial coefficients determined through numerical decomposition into the polynomial basic algebra model structure set in the previous stage, a camera global focal length error correction mapping relationship model with completely solidified parameters is formed. This solidified mapping relationship model can then be used to directly correct any input original nominal focal length value through a one-step fast calculation and output it as an extremely high-precision actual physical focal length value that fully conforms to the real optical physical projection relationship.

[0057] To verify the effectiveness of the multi-channel gradient field fusion scheme, a control experiment was conducted. The experimental conditions involved corner detection on a dataset of 50 checkerboard images covering both strong directional lighting and low-light scenarios. Control group one used a fusion method that averaged the gradient magnitudes of the RGB three channels, while control group two used a method that calculated the gradient based solely on the luminance channel. Experimental data showed that the corner detection repeatability of this invention was 94.5%, compared to 88.0% for control group one and 85.5% for control group two. The results demonstrate that this invention can stably detect the vast majority of corners under different lighting conditions. The improvement lies in the fact that the gradient direction entropy weighting assigns higher weights to color channels containing richer structural information and less noise, suppressing the negative impact of oversaturation or noise interference in a single channel.

[0058] To evaluate the performance of the scale-based corner detection and sub-pixel localization scheme, an ablation experiment was planned. The experimental conditions involved using a set of checkerboard images taken from far to near, varying the apparent scale of corner features from 3 pixels to 15 pixels. The baseline method used a fixed-scale Harris corner detector and a translation model without affine transformation for sub-pixel localization. The root mean square error (RMSE) of corner localization was recorded in the experiment. The RMS error of the baseline method was 0.45 pixels, while the RMS error of the proposed scheme was 0.08 pixels. Experimental results demonstrate that the proposed scheme has an accuracy advantage in multi-scale scenes. By determining the feature scale through the Laplacian response and using this feature scale to initialize the affine transformation matrix, the geometric deformation caused by perspective effects can be modeled, thus achieving sub-pixel localization accuracy superior to traditional methods.

[0059] To verify the contributions of weighted optimization and geometric regularization, a comparative experiment was conducted. The experiment used a sparse dataset containing only eight images to calibrate the zoom lens, some of which were slightly blurry. The control group employed a standard global optimization method, where the reprojection error weights were equal for all points and no regularization term was used. The experimental data were evaluated using average reprojection error and focal length estimation stability. The control group had an average reprojection error of 0.37 pixels and a standard deviation of 35.6 pixels for focal length estimation across repeated experiments. The present invention had an average reprojection error of 0.12 pixels and a standard deviation of 5.2 pixels for focal length estimation. The experimental results demonstrate that the present invention can still achieve accurate and stable calibration results even with insufficient or poor-quality data. The improvement lies in the fact that the positional confidence weight reduces the adverse effects of blurred or inaccurately positioned corner points on optimization, while the collinearity and vanishing point regularization terms provide strong geometric prior constraints for the optimization problem, preventing the optimization process from falling into an ill-conditioned solution space caused by sparse data.

[0060] While various embodiments of the invention have been shown and described in this specification, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention.

Claims

1. A method for zoom camera focal length error correction based on image processing, characterized in that, The method comprises the following steps: acquiring calibration board images captured by a zoom camera at multiple different zoom positions; for any calibration board image, calculating gradient direction entropy of each pixel in a neighborhood based on R, G and B channels, and generating a multi-channel gradient field fusion image with the reciprocal of the gradient direction entropy as the weight; constructing a Gaussian scale space pyramid on the multi-channel gradient field fusion image, calculating multi-scale Harris-Laplace responses, and performing non-maximum suppression in the scale space joint domain to obtain initial coordinates and feature scales of candidate calibration points; For each candidate calibration point, a two-dimensional gray scale template with fixed size representing the ideal corner pattern and a corresponding reference scale are constructed, a local neighborhood gray scale topological model with associated local illumination affine distortion and geometric projection size constraint related to the feature scale is established on the image layer corresponding to the feature scale, which expresses the image predicted gray scale value within the local neighborhood of the candidate calibration point as a function of the illumination affine parameters, the affine transformation matrix, the two-dimensional gray scale template and the sub-pixel coordinate offset of the calibration point, the expression of the function is: wherein, is the corresponding image predicted gray scale value, is the illumination contrast parameter, is the illumination intensity offset parameter, and constitute the illumination affine parameters, is the ideal gray scale value of the two-dimensional gray scale template at the corresponding coordinate position, is the horizontal template coordinate variable, is the vertical template coordinate variable; the image actual pixel coordinates and the two-dimensional gray scale template coordinates satisfy the coordinate correlation transformation equation and is the initial coordinate of the candidate calibration point, and are the sub-pixel coordinate offsets in the horizontal direction and the vertical direction respectively, is the affine transformation matrix; and are the horizontal coordinate and the vertical coordinate of any actual pixel point within the local neighborhood of the candidate calibration point respectively; the initial value of the determinant of the affine transformation matrix is constrained to be the square of the ratio of the feature scale to the reference scale according to the feature scale of the candidate calibration point, the sub-pixel coordinate offset is solved to update the sub-pixel coordinate of the candidate calibration point by minimizing the residual error between the image predicted gray scale value calculated by the local neighborhood gray scale topological model and the image actual gray scale value through an iterative algorithm, and the positioning confidence is calculated in inverse proportion to the residual error;​ alternately optimizing sub-pixel coordinates of all calibration points of all calibration board images and camera parameters at each zoom position until convergence; after the iterative convergence, extracting the nominal focal length and the actual focal length obtained by optimization at each zoom position, establishing a mapping relationship model between them, and completing focal length error correction.

2. The image processing based zoom camera focal length error correction method of claim 1, wherein, The method for obtaining the multi-channel gradient field fusion image comprises the following steps: for R, G and B channel images, calculating gradient amplitude and gradient direction of each pixel in each channel by using a 3*3 Sobel operator; for each pixel, representing the gradient direction as eight direction intervals in a 9*9 spatial neighborhood, and weighting and counting the gradient direction histogram of each channel according to the gradient amplitude; calculating the gradient direction entropy of each channel corresponding to each pixel according to the gradient direction histogram, extracting the reciprocal of the gradient direction entropy as the weight, and performing weighted average on each gradient amplitude of the three channels according to the weight corresponding to each channel to obtain a multi-channel gradient field fusion image.

3. The image processing based zoom camera focal length error correction method of claim 1, wherein, The method for obtaining initial coordinates and feature scales of candidate calibration points comprises the following steps: constructing the Gaussian scale space pyramid by using a set initial scale and a scale factor, and the Gaussian scale space pyramid comprises multiple groups and each group comprises multiple scale layers; on each scale layer, calculating a 2*2 autocorrelation matrix and a Harris corner response value, and the Harris corner response value is the determinant of the autocorrelation matrix minus the product of an empirical constant and the square of the trace of the autocorrelation matrix; in the scale dimension, determining the feature scale by calculating the extreme value of the Gaussian Laplace response value of each pixel; in the scale space joint domain, performing non-maximum suppression on the Harris corner response in a 3*3*3 neighborhood, and the maximum value point after the suppression is determined as a candidate calibration point, the two-dimensional spatial coordinates of the maximum value point are taken as the initial coordinates, and the scale value of the scale layer where the maximum value point is located is taken as the feature scale.

4. The image processing based zoom camera focal length error correction method of claim 1, wherein, The method for calculating the position confidence degree in inverse proportion to the residual error comprises the following steps: obtaining the minimum sum of squares of differences output by the iterative algorithm after convergence as a numerical residual error; dividing a constant 1 by the sum of the numerical residual error and a correction parameter for preventing the denominator from being zero to obtain an initial confidence degree; performing unified normalization processing on the initial confidence degrees of all candidate calibration points in a single calibration board image, and determining the normalized numerical value as the position confidence degree.

5. The image processing based zoom camera focal length error correction method of claim 1, wherein, The process of the alternate iterative optimization performs parameter updating and coordinate updating.

6. The image processing based zoom camera focal length error correction method of claim 5, wherein, The parameter updating is to fix the current sub-pixel coordinates of all candidate calibration points, update camera parameters under each zoom position based on a camera calibration model, and the camera parameters include focal length, principal point and distortion coefficients; the coordinate updating is to fix the updated camera parameters, and globally optimize the sub-pixel coordinates of all candidate calibration points by minimizing a joint energy function; The joint energy function is a sum of a weighted re-projection error term and a regularization term, and the calculation process of the weighted re-projection error term includes: projecting a theoretical position of the calibration board in a three-dimensional model coordinate system to a two-dimensional image plane by using the currently updated camera parameters to obtain theoretical projection point coordinates; calculating a norm square of a deviation distance between the theoretical projection point coordinates of each candidate calibration point and the currently updated sub-pixel coordinates to obtain a re-projection error; and performing numerical weighted summation on the re-projection error of each candidate calibration point by using a corresponding calibration position confidence to obtain the weighted re-projection error term; The calculation process of the regularization term includes: for the projection coordinates of any three calibration points on the same physical straight line on the calibration board image, calculating a geometric distance from the middle projection coordinate to the straight line determined by the other two projection coordinates, and accumulating a square sum of all such geometric distances as a collinear constraint energy term; for a plurality of sets of parallel straight lines on the calibration board image, respectively establishing corresponding intersection points as vanishing points in a homogeneous coordinate system, and calculating a square sum of inner products of homogeneous algebraic vectors of each set of straight lines and a homogeneous coordinate vector of the corresponding vanishing point as a vanishing point constraint energy term; and summing the collinear constraint energy term and the vanishing point constraint energy term by a preset weighting coefficient to constitute the regularization term.

7. The image processing based zoom camera focal length error correction method of claim 1, wherein, The method includes the following steps: extracting the nominal focal lengths and the optimized actual focal lengths under each zoom position, establishing a mapping relationship model between the two, and completing focal length error correction, including: extracting paired data of the nominal focal lengths and the actual focal lengths under all zoom positions; setting a polynomial model to fit the mapping relationship between the nominal focal lengths and the actual focal lengths; using the least square method to solve polynomial coefficients that make the fitting error square sum of all paired data minimum; fixing the solved polynomial coefficients to be used for correcting any input nominal focal length value to a corresponding actual focal length value.

Images