Method and system for calibrating preset position of PTZ (Pan / Tilt / Zoom) camera

Abstract

The invention discloses a method and a system for calibrating a preset position of a pan-tilt camera, belongs to the technical field of computer vision and image processing, and solves the problem that full-process automation and multi-dimensional coordination of image definition optimization, precise pose adjustment and focal length and size calibration cannot be realized in the prior art. According to the method, on the basis of the cradle head camera with the currently set focal length and pose, a gradient-based definition evaluation algorithm is adopted to carry out definition judgment on an original image collected by the cradle head camera so as to obtain a clear image; s2, carrying out feature point matching on the clear image and the standard image, if matching fails, controlling the pan-tilt camera to rotate, executing the step S1 again, and if matching succeeds, carrying out homography matrix solving based on the feature points obtained through matching so as to guide the pan-tilt camera to carry out pose calibration; and after the pose calibration, obtaining a new image to carry out zoom coefficient calculation, and carrying out focal length adjustment according to the zoom coefficient so as to complete final calibration. The method is used for calibrating the preset position of the pan-tilt camera.

Description

TECHNICAL FIELD

[0001] The application relates to a gimbal camera preset position calibration method and system, and belongs to the technical field of computer vision and image processing. BACKGROUND

[0002] In an intelligent early warning system of a transformer substation, a gimbal camera is usually used to photograph multiple targets. For different targets, the gimbal camera is required to have different poses and focal lengths. The existing system usually records the pose and focal length of the gimbal camera corresponding to each target, and controls the gimbal camera to reach the specified pose and focal length when photographing the corresponding target. After a long time of operation, the gimbal camera often cannot move to the specified pose due to reasons such as back error and motor step loss, so that the target image cannot be correctly collected.

[0003] Accurate shooting position calibration is the key to ensuring the imaging accuracy of the gimbal camera. Existing calibration technologies mainly include the following two types:

[0004] Calibration technology based on external sensors: the distance information of a target object is obtained through a radar, a laser range finder and the like, and the camera pose offset is calculated by combining a geometric model; a magnetic angle sensor and an IMU (inertial measurement unit) are used to monitor the rotation angle of the gimbal in real time, and a fuzzy PID algorithm is used to compensate the angle error caused by the motor gap.

[0005] Calibration technology based on image analysis: image feature points are manually selected and compared with standard images, and the pose and focal length of the gimbal are manually adjusted (traditional manual calibration method); a monocular vision or multi-view vision algorithm is used to calculate the relative pose of the camera, but usually only the angle deviation is calibrated, and there is a lack of comprehensive optimization of the focal length and the image size.

[0006] In summary, the existing technology has the following technical problems:

[0007] 1. Strong hardware dependency, high cost and complexity: the scheme depending on external sensors such as radars and laser range finders needs to deploy special hardware, increasing the cost and complexity of installation; and for the gimbal cameras that have been put into use, if the main control board or sensor hardware needs to be modified, it is difficult to realize technical updates, and the compatibility is poor;

[0008] 2. Single calibration dimension, insufficient accuracy: the scheme based on sensors only compensates the mechanical error of the motor through angle compensation, without involving automatic optimization of image clarity and focal length, and cannot ensure the imaging quality; the scheme based on image analysis can calculate the relative pose, but lacks calibration of image size consistency (such as image scaling error caused by focal length change), and cannot meet the demand for pixel-level accuracy in industrial detection and surveying.

[0009] 3. Low degree of automation, relying on manual intervention: the traditional manual calibration method requires the operator to manually adjust the focal length and the contrast image features, which is inefficient and easily affected by subjective judgment; some semi-automatic solutions require the user to actively trigger the calibration process and input initial parameters, which cannot achieve full-process autonomous calibration and is difficult to adapt to unattended scenarios (such as outdoor monitoring and automatic inspection of unmanned aerial vehicles).

[0010] 4. Poor adaptability to complex scenes: in complex environments such as occlusion, light variation, and feature point loss, existing image matching algorithms (such as solutions relying only on SIFT features) are prone to matching failure, and lack effective coping strategies. SUMMARY

[0011] To solve the problem that the prior art cannot achieve full-process automation and multi-dimensional coordination of "image sharpness optimization-pose precision adjustment-focal length and size calibration", resulting in difficulty in meeting the high-end application requirements in terms of calibration efficiency, accuracy, and scene adaptability, the present application aims to provide a gimbal camera preset position calibration method and system.

[0012] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0013] A gimbal camera preset position calibration method, comprising the following steps:

[0014] Step S1, based on the current set focal length and pose of the gimbal camera, a sharpness evaluation algorithm based on gradient is used to perform sharpness judgment on the original image of the target scene collected by the gimbal camera, to obtain a clear image;

[0015] Step S2, feature point matching is performed between the clear image and the standard image, if the matching fails, the gimbal camera is controlled to rotate, and step S1 is executed again, if the matching succeeds, homography matrix solving is performed based on the feature points obtained by matching, to guide the gimbal camera to perform pose calibration;

[0016] Step S3, after pose calibration, new images are obtained for scaling coefficient calculation, and focal length adjustment is performed according to the scaling coefficient, to complete the final calibration.

[0017] Further, the specific steps of step S1 are as follows:

[0018] Step S1.1, when the gimbal camera receives an image acquisition instruction, the original image of the target scene is collected based on the current set focal length and pose;

[0019] Step S1.2, the original image is converted into a gray-scale image, and the gradient amplitude of each pixel point in the gray-scale image is calculated using the Sobel operator or the Prewitt operator, wherein for a pixel point with coordinates (i, j) in the gray-scale image, its pixel value is f (i，j)Calculate its horizontal gradient G x (i, j) and the vertical gradient G y (i, j), and based on the directional gradient G x (i, j) and the vertical gradient G y The gradient magnitude of the pixel at coordinates (i, j) is calculated using the following formula:

[0020]

[0021] G x (i, j) = f ((i+1，j+1)) +2f ((i，j+1)) +f ((i-1，j+1)) -f ((i-1，j-1)) -2f ((i，j-1)) -f ((i+1，j-1))

[0022] G y (i, j) = f ((i-1，j-1)) +2f ((i-1，j)) +f ((i-1，j+1)) -f ((i+1，j-1)) -2f ((i+1，j)) -f ((i+1，j+1)) Mean and variance of gradient magnitudes based on grayscale images:

[0023]

[0024] Where M is the mean of the gradient magnitude of the grayscale image, V is the variance of the gradient magnitude of the grayscale image, and the variance is used as the sharpness score. The larger the variance, the sharper the original image. N represents the number of pixels in the grayscale image.

[0025] Step S1.3: If the variance is lower than the preset sharpness threshold, the original image is determined to be blurry. At this time, the control module of the gimbal camera sends a command to its focus adjustment mechanism to adjust the direction, and the original image is re-acquired and the steps are executed again.

[0026] S1.2, otherwise, a clear image is obtained.

[0027] Furthermore, the specific steps of step S2 are as follows:

[0028] Step S2.1: Use a feature point matching algorithm to match the clear image with the pre-stored standard image to obtain preliminary feature point pairs. Then filter out feature point pairs that do not meet the epipolar constraint to obtain the final image matching feature point pairs.

[0029] Step S2.2: If the number of feature point pairs matched in the final image is less than the preset value, it is determined that the match is incorrect. At this time, the control module controls the gimbal camera to move in the horizontal and vertical directions in sequence to gradually adjust the gimbal pose. After adjustment, step 1 is executed again. Otherwise, the match is successful, that is, the scene captured by the clear image and the standard image is the same scene, and step S2.3 is executed.

[0030] Step S2.3: Calculate the relative pose of the final image matching feature point pairs based on the clear image and the standard image, specifically as follows:

[0031] Let the intrinsic parameter matrix K of the gimbal camera be:

[0032]

[0033] Among them, f u and f v Let f0 and v0 be the equivalent pixel focal lengths in the X and Y directions, respectively, and u0 and v0 be the X-axis and Y-axis coordinates of the principal pixel of the gimbal camera, respectively. Since the scaling factor in the horizontal and vertical directions is usually equal during the manufacturing process of a gimbal camera, i.e., f0... u =f v =s, where s is the overall scaling factor, and K is:

[0034]

[0035] Using the camera coordinate system of the standard image as the world coordinate system, let the direction vector of the projection from any point in the world coordinate system onto the gimbal camera coordinate system be P = [a, b, c]. T Let a, b, c represent the projections of the direction vector P onto the X, Y, and Z axes, respectively. (a, b, c) satisfy a 2 +b 2 +c 2 =1, a<<1, b<<1, c≈1, the coordinates (u, v) of this point projected onto the pixel plane are:

[0036]

[0037] After the gimbal camera rotates relative to the standard image by angles φ and θ around the X and Y axes respectively, the current image to be matched is obtained. The projection direction vector of the same point becomes P′. The transformation process is as follows:

[0038]

[0039] When photographing a distant target, if the target appears in both the standard image and the current image to be matched, then θ << 1 and φ << 1. In this case, sinθ ≈ θ, sinφ ≈ φ, cosθ ≈ 1, and cosφ ≈ 1. P′ is approximately:

[0040]

[0041] Ignoring second-order minterms aθφ, bφ, and aθ:

[0042]

[0043] The coordinates (u′, v′) of P′ projected onto the pixel plane are:

[0044]

[0045] The pixel offset between the current image to be matched and the standard image is:

[0046]

[0047] At this point, only translation and scaling exist between the standard image and the current image to be matched. The homography matrix between the two images is equivalent to:

[0048]

[0049] The pixel offset is proportional to the rotation angle of the gimbal camera. By adjusting the angle of the gimbal camera, the pixel offset is made approximately equal to 0, so that the principal points of the two images coincide.

[0050] The relative pose is calculated using a homography matrix-based method and the final image matching feature point pairs. Specifically:

[0051] Let the sharp image be I1 and the standard image be I2. The coordinates of the final image matching feature point pairs obtained from the sharp image I1 and the standard image I2 are respectively... and n is the final number of feature point pairs matched in the image.

[0052] Let the homography matrix H of the sharp image (I1) and the standard image (I2) under any pose be defined as:

[0053]

[0054] The mapping relationship between feature matching points of the clear image (I1) and the standard image (I2) is defined as follows:

[0055]

[0056] Where k is the scaling factor, and the final feature matching point pairs in the two images, I1 (sharp image) and I2 (standard image), have the following relationship with the homography matrix:

[0057]

[0058] Through N groups and The final feature matching point pairs are used to list 2N linear equations, and the elements in H are solved using the least squares method, where N > 8;

[0059] Since at this point, the only differences between the clear image I1 and the standard image I2 are jitter and scaling, H 12 H 21 H 31 and H 32 The homography matrix H between the clear image I1 and the standard image I2 is approximately 0. ts :

[0060] H≈H ts

[0061]

[0062] The homography matrix H obtained by solving the relationship between the final feature matching point pairs between the clear image I1 and the standard image I2 and the homography matrix satisfies:

[0063]

[0064] Among them, H 11 and H 22 The relative scaling factors in the X and Y directions between the sharp image I1 and the standard image I2 are typically taken as their average. This average is used to guide the gimbal camera in adjusting the focus. 13 and H 23 These are the pixel offsets in the X and Y directions between the clear image I1 and the standard image I2, respectively. These two values ​​are used to guide the movement of the gimbal camera and perform pose alignment.

[0065] In this step, via H 13 and H 23 Guide the gimbal camera to move until the real-time camera pose coincides with the camera pose of the standard image, and after coincidence, satisfy H 13 ≈0, H 23 ≈0.

[0066] Furthermore, the specific steps of step S3 are as follows:

[0067] Step 3.1: After pose calibration, when the pan-tilt camera's shooting position coincides with the standard image shooting position, there is no relative offset between the two shooting positions, only relative scaling. The homography matrix is ​​equivalent to:

[0068]

[0069] New images are acquired again, and feature point pair matching and homography matrix calculation are performed sequentially. The scaling factor is then obtained based on the homography matrix.

[0070]

[0071] Step 3.2: If the scaling factor is not 1, the control module of the gimbal camera adjusts the focal length according to the difference between the scaling factor and 1. After adjusting the focal length, the image is re-acquired and step 3.1 is executed again. Otherwise, the scaling factor s satisfies |s-1|<∈, where ∈ is a minimum threshold, and the final calibration is completed.

[0072] A system for calibrating a preset position of a gimbal camera includes:

[0073] Image acquisition and sharpness assessment module: Based on the currently set focal length and pose of the gimbal camera, a gradient-based sharpness assessment algorithm is used to judge the sharpness of the original images of the target scene acquired by the gimbal camera in order to obtain a sharp image.

[0074] Matching and pose calibration module: Match the clear image with the standard image by feature points. If the matching fails, control the gimbal camera to rotate and execute step S1 again. If the matching is successful, solve the homography matrix based on the matched feature points to guide the gimbal camera to perform pose calibration.

[0075] After pose calibration, a new image is acquired, the scaling factor is calculated, and the focal length is adjusted according to the scaling factor to complete the final calibration.

[0076] Control module: Controls the gimbal camera based on instructions from the image acquisition and sharpness assessment module and the matching and pose calibration module.

[0077] Furthermore, the specific implementation steps of the image acquisition and sharpness assessment module are as follows:

[0078] Step S1.1: When the gimbal camera receives the image acquisition command, it acquires the original image of the target scene based on the currently set focal length and pose.

[0079] Step S1.2: Convert the original image to a grayscale image, and calculate the gradient magnitude of each pixel in the grayscale image using the Sobel operator or the Prevet operator. For the pixel at coordinates (i, j) in the grayscale image, its pixel value is f. (i，j) Calculate its horizontal gradient G x (i, j) and the vertical gradient G y (i, j), and based on the directional gradient G x (i, j) and the vertical gradient G y The gradient magnitude of the pixel at coordinates (i, j) is calculated using the following formula:

[0080]

[0081] Gx (i, j) = f ((i+1，j+1)) +2f ((i，j+1)) +f ((i-1，j+1)) -f ((i-1，j-1)) -2f ((i，j-1)) -f ((i+1，j-1))

[0082] G y (i, j) = f ((i-1，j-1)) +2f ((i-1，j)) +f ((i-1，j+1)) -f ((i+1，j-1)) -2f ((i+1，j)) -f ((i+1，j+1))

[0083] Mean and variance of gradient magnitudes based on grayscale images:

[0084]

[0085] Where M is the mean of the gradient magnitude of the grayscale image, V is the variance of the gradient magnitude of the grayscale image, and the variance is used as the sharpness score. The larger the variance, the sharper the original image. N represents the number of pixels in the grayscale image.

[0086] Step S1.3: If the variance is lower than the preset sharpness threshold, the original image is determined to be blurry. At this time, the control module of the gimbal camera sends a command to its focus adjustment mechanism to adjust the direction, and the original image is re-acquired and the steps are executed again.

[0087] S1.2, otherwise, a clear image is obtained.

[0088] Furthermore, the specific implementation steps of pose calibration in the matching and pose calibration module are as follows:

[0089] Step S2.1: Use a feature point matching algorithm to match the clear image with the pre-stored standard image to obtain preliminary feature point pairs. Then filter out feature point pairs that do not meet the epipolar constraint to obtain the final image matching feature point pairs.

[0090] Step S2.2: If the number of feature point pairs matched in the final image is less than the preset value, it is determined that the match is incorrect. At this time, the control module controls the gimbal camera to move in the horizontal and vertical directions in sequence to gradually adjust the gimbal pose. After adjustment, step 1 is executed again. Otherwise, the match is successful, that is, the scene captured by the clear image and the standard image is the same scene, and step S2.3 is executed.

[0091] Step S2.3: Calculate the relative pose of the final image matching feature point pairs based on the clear image and the standard image, specifically as follows:

[0092] Let the intrinsic parameter matrix K of the gimbal camera be:

[0093]

[0094] Among them, f u and f v Let f0 and v0 be the equivalent pixel focal lengths in the X and Y directions, respectively, and u0 and v0 be the X-axis and Y-axis coordinates of the principal pixel of the gimbal camera, respectively. Since the scaling factor in the horizontal and vertical directions is usually equal during the manufacturing process of a gimbal camera, i.e., f0... u =f v =s, where s is the overall scaling factor, and K is:

[0095]

[0096] Using the camera coordinate system of the standard image as the world coordinate system, let the direction vector of the projection from any point in the world coordinate system onto the gimbal camera coordinate system be P = [a, b, c]. T Let a, b, c represent the projections of the direction vector P onto the X, Y, and Z axes, respectively. (a, b, c) satisfy a 2 +b 2 +c 2 =1, a<<1, b<<1, c≈1, the coordinates (u, v) of this point projected onto the pixel plane are:

[0097]

[0098] After the gimbal camera rotates relative to the standard image by angles φ and θ around the X-axis and T-axis respectively, the current image to be matched is obtained. The projection direction vector of the same point becomes P′. The transformation process is as follows:

[0099]

[0100] When photographing a distant target, if the target appears simultaneously in both the standard image and the current image to be matched, then θ << 1 and φ << 1. In this case, sinθ ≈ θ, sinφ ≈ φ, cosθ ≈ 1, and Cosφ ≈ 1. P′ is approximately:

[0101]

[0102] After ignoring the second-order minterms aθφ, bφ, and aθ:

[0103]

[0104] The coordinates (u′, v′0) of P′ projected onto the pixel plane are:

[0105]

[0106] The pixel offset between the current image to be matched and the standard image is:

[0107]

[0108] At this point, only translation and scaling exist between the standard image and the current image to be matched. The homography matrix between the two images is equivalent to:

[0109]

[0110] The pixel offset is proportional to the rotation angle of the gimbal camera. By adjusting the angle of the gimbal camera, the pixel offset is made approximately equal to 0, so that the principal points of the two images coincide.

[0111] The relative pose is calculated using a homography matrix-based method and the final image matching feature point pairs. Specifically:

[0112] Let the sharp image be I1 and the standard image be I2. The coordinates of the final image matching feature point pairs obtained from the sharp image I1 and the standard image I2 are respectively... and n is the final number of image matching feature point pairs. Let the homography matrix H of the sharp image (I1) and the standard image (I2) under any pose be defined as:

[0113]

[0114] The mapping relationship between feature matching points of the clear image (I1) and the standard image (I2) is defined as follows:

[0115]

[0116] Where k is the scaling factor, and the final feature matching point pairs in the two images, I1 (sharp image) and I2 (standard image), have the following relationship with the homography matrix:

[0117]

[0118] Through N groups and The final feature matching point pairs are used to list 2N linear equations, and the elements in H are solved using the least squares method, where N > 8;

[0119] Since at this point, the only differences between the clear image I1 and the standard image I2 are jitter and scaling, H 12 H 21 H 31 and H 32 The homography matrix H between the clear image I1 and the standard image I2 is approximately 0. ts :

[0120] H≈H ts

[0121]

[0122] The homography matrix H obtained by solving the relationship between the final feature matching point pairs between the clear image I1 and the standard image I2 and the homography matrix satisfies:

[0123]

[0124] Among them, H 11 and H 22 The relative scaling factors in the X and Y directions between the sharp image I1 and the standard image I2 are typically taken as their average. This average is used to guide the gimbal camera in adjusting the focus. 13 and H 23 These are the pixel offsets in the X and Y directions between the clear image I1 and the standard image I2, respectively. These two values ​​are used to guide the movement of the gimbal camera and perform pose alignment.

[0125] In this step, via H 13 and H 23 Guide the gimbal camera to move until the real-time camera pose coincides with the camera pose of the standard image, and after coincidence, satisfy H 13 ≈0, H 23 ≈0.

[0126] Furthermore, the specific steps in the matching and pose calibration module, after pose calibration, to acquire a new image, calculate the scaling factor, and adjust the focal length according to the scaling factor to complete the final calibration, are as follows:

[0127] Step 3.1: After pose calibration, when the pan-tilt camera's shooting position coincides with the standard image shooting position, there is no relative offset between the two shooting positions, only relative scaling. The homography matrix is ​​equivalent to:

[0128]

[0129] New images are acquired again, and feature point pair matching and homography matrix calculation are performed sequentially. The scaling factor is then obtained based on the homography matrix.

[0130]

[0131] Step 3.2: If the scaling factor is not 1, the control module of the gimbal camera adjusts the focal length according to the difference between the scaling factor and 1. After adjusting the focal length, the image is re-acquired and step 3.1 is executed again. Otherwise, the scaling factor s satisfies |s-1|<∈, where ∈ is a minimum threshold, and the final calibration is completed.

[0132] Compared with the prior art, the beneficial effects of this invention are as follows:

[0133] I. An automated image sharpness and focus optimization mechanism enables full-process automation and reduces manual intervention.

[0134] This invention uses a gradient-based sharpness evaluation algorithm to evaluate image sharpness in real time and automatically adjust the focal length until a sharpness threshold is reached. This process requires no manual intervention and can quickly adapt to the focal length requirements of different shooting scenarios, ensuring image quality in the input matching and pose calculation stages. It improves the autonomy and efficiency of the calibration process. In other words, this invention automatically and cyclically executes everything from image acquisition, sharpness judgment, pose adjustment to focal length optimization without manual operation, improving the timeliness and efficiency of calibration and reducing labor costs.

[0135] II. Handling Image Matching Failures

[0136] When the number of image matching feature point pairs is insufficient, the present invention provides full field-of-view scanning to ensure that a field of view that overlaps with the standard image can be found, thereby enabling subsequent pose calibration and increasing the probability of successful camera preset position calibration.

[0137] III. Precise image size calibration based on scaling factor, with more comprehensive calibration dimensions.

[0138] This invention calculates the scaling factor between a sharp image and a standard image, and fine-tunes the focal length to make the scaling factor approach 1, ensuring that the scale of the final captured image is consistent with that of the standard image. In other words, this invention not only ensures accurate shooting position through image matching and pose calculation, but also optimizes the focal length through sharpness judgment and uses the scaling factor to achieve precise calibration of image size. It ensures imaging accuracy from multiple dimensions such as position, focal length, and image size, and the calibration is more comprehensive.

[0139] IV. The closed-loop calibration process design makes it highly adaptable to complex scenarios.

[0140] This invention, from image sharpness assessment and pose adjustment to focus optimization, incorporates feedback and cyclic adjustment mechanisms (such as focus adjustment loops and pose coincidence assessment loops) at each stage, forming a complete closed-loop calibration process. This continuously optimizes calibration results, minimizing errors and improving calibration stability and accuracy. Specifically, this invention provides a full-field scanning function for image matching failures. Combined with the closed-loop process design (such as focus adjustment loops and pose coincidence assessment loops), it can achieve calibration through multiple iterations in complex environments (such as occlusion and lighting changes). Existing technologies (such as those relying on fixed sensor data or single compensation strategies) have limited ability to handle complex image-level changes. This invention, through image feature analysis and multi-strategy adjustments, offers superior adaptability.

[0141] V. No additional dedicated hardware required, wide applicability

[0142] Existing technologies (such as those relying on radar or magnetic angle sensors) have specific hardware requirements, increasing cost and complexity. This invention achieves calibration based on image analysis (sharpness calculation, feature matching, and scaling factor adjustment), requiring no additional hardware. It is applicable to various gimbal cameras, especially deployed equipment, and has stronger compatibility and scalability. Attached Figure Description

[0143] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0144] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0145] A gimbal camera shooting position calibration solution that requires no additional hardware, covers multiple calibration dimensions, and has autonomous closed-loop adjustment capabilities is proposed to address the shortcomings of existing technologies.

[0146] This invention calculates the camera's motion relationship between continuously acquired images and images at preset positions, thereby controlling the camera to reach a designated preset position and complete the target capture. Specifically:

[0147] A method for calibrating a preset position of a gimbal camera includes the following steps:

[0148] Step S1: After the system starts up, or when an external calibration command is received, the control module sends an image acquisition command to the gimbal camera. The gimbal camera, based on the currently set focal length and pose, uses a gradient-based sharpness evaluation algorithm to judge the sharpness of the original image of the target scene acquired by the gimbal camera in order to obtain a sharp image.

[0149] The specific steps are as follows:

[0150] Step S1.1: When the gimbal camera receives the image acquisition command, it acquires the original image of the target scene based on the currently set focal length and pose.

[0151] Step S1.2: Convert the original image to a grayscale image, and calculate the gradient magnitude of each pixel in the grayscale image using the Sobel operator or the Prevet operator. For the pixel at coordinates (i, j) in the grayscale image, its pixel value is f. (i，j) Calculate its horizontal gradient G x (i, j) and the vertical gradient G y (i, j), and based on the directional gradient G x (i, j) and the vertical gradient G y The gradient magnitude of the pixel at coordinates (i, j) is calculated using the following formula:

[0152]

[0153] G x (i, j) = f ((i+1，j+1)) +2f ((i,j+1)) +f ((i-1，j+1)) -f ((i-1，j-1)) -2f ((i，j-1)) -f ((i+1，j-1))

[0154] G y (i, j) = f ((i-1，j-1)) +2f ((i-1，j)) +f ((i-1，，j+1)) -f ((i+1，j-1)) -2f ((i+1，j)) -f ((i+1，j+1))

[0155] Mean and variance of gradient magnitudes based on grayscale images:

[0156]

[0157] Where M is the mean of the gradient magnitude of the grayscale image, V is the variance of the gradient magnitude of the grayscale image, and the variance is used as the sharpness score. The larger the variance, the sharper the original image. N represents the number of pixels in the grayscale image.

[0158] Step S1.3: If the variance is lower than the preset clarity threshold (such as an empirical value determined through experiments), the original image is judged to be unclear. At this time, the control module of the gimbal camera sends a command to its focal length adjustment mechanism (such as a lens focusing device driven by a stepper motor) to adjust the direction, that is, adjust in a fixed step size (such as adjusting the focal length by 0.1 mm each time) in the direction of increasing clarity, and re-acquire the original image and execute step S1.2 again. Otherwise, a clear image is obtained.

[0159] Step S2: Match the clear image with the standard image for feature points. If the matching fails, control the gimbal camera to rotate and execute step S1 again. If the matching succeeds, solve the homography matrix based on the matched feature points to guide the gimbal camera to perform pose calibration.

[0160] The specific steps are as follows:

[0161] Step S2.1: Use a feature point matching algorithm (such as ORB, SIFT, SURF, Efficient LoFTR, etc.) to match the clear image with the pre-stored standard image to obtain preliminary feature point pairs. Then filter out feature point pairs that do not meet the epipolar constraint to obtain the final image matching feature point pairs.

[0162] Step S2.2: If the number of feature point pairs matched in the final image is less than the preset value (e.g., 8 pairs), it is determined that the match is incorrect. At this time, the control module controls the gimbal camera to move in the horizontal and vertical directions in sequence to gradually adjust the gimbal pose. After adjustment, step 1 is executed again. Otherwise, the match is successful, that is, the scene captured by the clear image and the standard image is the same scene, and step S2.3 is executed.

[0163] Step S2.3: Calculate the relative pose of the final image matching feature point pairs based on the clear image and the standard image, specifically as follows:

[0164] Let the intrinsic parameter matrix K of the gimbal camera be:

[0165]

[0166] Among them, f u and f v Let f0 and v0 be the equivalent pixel focal lengths in the X and Y directions, respectively. u0 and v0 are the X-axis and T-axis coordinates of the principal pixel of the gimbal camera, respectively. Since the scaling factor in the horizontal and vertical directions is usually equal during the manufacturing process of a gimbal camera, i.e., f0... u =f v =s, where s is the overall scaling factor, and K is:

[0167]

[0168] Using the camera coordinate system of the standard image as the world coordinate system, let the direction vector of the projection from any point in the world coordinate system onto the gimbal camera coordinate system be P = [a, b, c]. T Let a, b, c represent the projections of the direction vector P onto the X, Y, and Z axes, respectively. (a, b, c) satisfy a 2 +b 2 +c 2 =1, a<<1, b<<1, c≈1, the coordinates (u, v) of this point projected onto the pixel plane are:

[0169]

[0170] After the gimbal camera rotates relative to the standard image by angles φ and θ around the X and Y axes respectively, the current image to be matched is obtained. The projection direction vector of the same point becomes P′. The transformation process is as follows:

[0171]

[0172] When photographing a distant target, if the target appears simultaneously in both the standard image and the current image to be matched, then θ << 1 and φ << 1. In this case, sinθ ≈ θ, sinφ ≈ φ, cosθ ≈ 1, and Cosφ ≈ 1. P′ is approximately:

[0173]

[0174] After ignoring the second-order minterms aθφ, bφ, and aθ:

[0175]

[0176] The coordinates (u′, v′) of P′ projected onto the pixel plane are:

[0177]

[0178] The pixel offset between the current image to be matched and the standard image is:

[0179]

[0180] At this point, only translation and scaling exist between the standard image and the current image to be matched. The homography matrix between the two images is equivalent to:

[0181]

[0182] The pixel offset is proportional to the rotation angle of the gimbal camera. By adjusting the angle of the gimbal camera, the pixel offset is made approximately equal to 0, so that the principal points of the two images coincide.

[0183] The relative pose is calculated using a homography matrix-based method and the final image matching feature point pairs. Specifically:

[0184] Let the sharp image be I1 and the standard image be I2. The coordinates of the final image matching feature point pairs obtained from the sharp image I1 and the standard image I2 are respectively... and n is the final number of feature point pairs matched in the image.

[0185] Let the homography matrix H of the sharp image (I1) and the standard image (I2) under any pose be defined as:

[0186]

[0187] The mapping relationship between feature matching points of the clear image (I1) and the standard image (I2) is defined as follows:

[0188]

[0189] Where k is the scaling factor, and the final feature matching point pairs in the two images, I1 (sharp image) and I2 (standard image), have the following relationship with the homography matrix:

[0190]

[0191] Through N groups and The final feature matching point pairs are used to list 2N linear equations, and the elements in H are solved using the least squares method, where N > 8;

[0192] Since at this point, the only differences between the clear image I1 and the standard image I2 are jitter and scaling, H 12 H 21 H 31 and H 32 The homography matrix H between the clear image I1 and the standard image I2 is approximately 0. ts :

[0193] H≈H ts

[0194]

[0195] The homography matrix H obtained by solving the relationship between the final feature matching point pairs between the clear image I1 and the standard image I2 and the homography matrix satisfies:

[0196]

[0197] Among them, H 11 and H 22 The relative scaling factors in the X and T directions between the sharp image I1 and the standard image I2 are typically taken as their average. This average is used to guide the gimbal camera in adjusting the focus. 13 and H 23 These are the pixel offsets in the X and T directions between the clear image I1 and the standard image I2, respectively. These two values ​​are used to guide the movement of the gimbal camera and perform pose alignment.

[0198] In this step, via H 13 and H 23 Guide the gimbal camera movement, set a position overlap threshold (e.g., pixel deviation less than 5 pixels), predict the corresponding position of the gimbal camera in the standard image based on the calculated relative pose, and compare the deviation between the actual acquired image and the predicted position. If the deviation is greater than the threshold, it is determined that the shooting positions do not overlap, the control module adjusts the gimbal pose (rotates the gimbal in the direction of reducing deviation), re-acquires the image, and repeats the above pose calculation and judgment process. If the deviation is less than the threshold, it is determined that the principal points of the two images coincide, and step S3 is performed to adjust the focal length.

[0199] Step S3: After pose calibration, acquire a new image, calculate the scaling factor, and adjust the focal length according to the scaling factor to complete the final calibration.

[0200] The specific steps are as follows:

[0201] Step 3.1: After pose calibration, when the pan-tilt camera's shooting position coincides with the standard image shooting position, there is no relative offset between the two shooting positions, only relative scaling. The homography matrix is ​​equivalent to:

[0202]

[0203] New images are acquired again, and feature point pair matching and homography matrix calculation are performed sequentially. The scaling factor is then obtained based on the homography matrix.

[0204]

[0205] Step 3.2: If the scaling factor is not 1, the control module of the gimbal camera adjusts the focal length according to the difference between the scaling factor and 1. After adjusting the focal length, the image is re-acquired and Step 3.1 is executed again. If s > 1 (image magnification), the focal length is reduced to shrink the image; if s < 1 (image shrinkage), the focal length is increased to magnify the image. Otherwise, the scaling factor s satisfies |s-1| < ∈, where ∈ is a very small threshold, such as 0.01, which completes the final calibration.

[0206] This invention presents a fully automatic focal length optimization method based on gradient variance and closed-loop control. It utilizes the Sobel / Prevet operator to calculate the image gradient magnitude variance as a sharpness score, and then adjusts the focal length in fixed steps (e.g., 0.1 mm) in the direction of increasing sharpness using a stepper motor, following a closed-loop control process until the variance is maximized. This method automatically optimizes image sharpness without manual intervention, providing a foundation for subsequent pose matching.

[0207] A relative pose calculation method based on epipolar constraints and image matching is proposed: Feature point pairs are extracted using algorithms such as ORB / SIFT / SURF / EfficientLoFTR, and false matches are filtered out using epipolar constraints. A mathematical model for calculating the gimbal rotation angle is based on the homography matrix H. This improves the robustness of matching in complex scenes by establishing a quantitative relationship between pixel offset and gimbal angle.

[0208] The phased, progressive calibration process control logic employs a closed-loop calibration process executed in stages: "sharpness optimization → pose alignment → focal length matching." It utilizes a pose coincidence threshold (e.g., pixel deviation less than 5 pixels) and a scaling factor threshold |s-1|<∈. This achieves dual calibration of position and scale, avoiding calibration errors caused by multi-parameter coupling.

[0209] A multi-module collaborative gimbal camera calibration system architecture: The software system comprises image acquisition, sharpness assessment, matching, pose calculation, and control modules; the hardware architecture integrates a high-precision servo gimbal with a motorized focusing camera. Deep software and hardware synergy enables fully automated calibration and compatibility with existing commercial gimbal camera systems.

[0210] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A method for calibrating a preset position of a gimbal camera, characterized in that, Includes the following steps: Step S1: Based on the currently set focal length and pose of the gimbal camera, a gradient-based sharpness evaluation algorithm is used to judge the sharpness of the original image of the target scene acquired by the gimbal camera in order to obtain a sharp image. Step S2: Match the clear image with the standard image for feature points. If the matching fails, control the gimbal camera to rotate and execute step S1 again. If the matching succeeds, solve the homography matrix based on the matched feature points to guide the gimbal camera to perform pose calibration. Step S3: After pose calibration, acquire a new image, calculate the scaling factor, and adjust the focal length according to the scaling factor to complete the final calibration.

2. The method for calibrating a preset position of a gimbal camera according to claim 1, characterized in that, The specific steps of step S1 are as follows: Step S1.1: When the gimbal camera receives the image acquisition command, it acquires the original image of the target scene based on the currently set focal length and pose. Step S1.2: Convert the original image to a grayscale image, and calculate the gradient magnitude of each pixel in the grayscale image using the Sobel operator or the Prevet operator. For the pixel at coordinates (i, j) in the grayscale image, its pixel value is f. (i，j) Calculate its horizontal gradient G x (i, j) and the vertical gradient G y (i, j), and based on the directional gradient G x (i, j) and the vertical gradient G y The gradient magnitude of the pixel at coordinates (i, j) is calculated using the following formula: G x (i, j)=f ((i+1，j+1)) +2f ((i,j+1)) +f ((i-1，j+1)) -f ((i-1，j-1)) -2f ((i，j-1)) -f ((i+1，j-1)) G y (i, j)=f ((i-1，j-1)) +2f ((i-1，j)) +f ((i-1，j+1)) -f ((i+1，j-1)) -2f ((i+1，j)) -f ((i+1，j+1)) Mean and variance of gradient magnitudes based on grayscale images: Where M is the mean of the gradient magnitude of the grayscale image, V is the variance of the gradient magnitude of the grayscale image, and the variance is used as the sharpness score. The larger the variance, the sharper the original image. N represents the number of pixels in the grayscale image. Step S1.3: If the variance is lower than the preset clarity threshold, the original image is determined to be unclear. At this time, the control module of the gimbal camera sends a command to its focus adjustment mechanism to adjust the direction, and the original image is re-acquired and step S1.2 is executed again. Otherwise, a clear image is obtained.

3. The method for calibrating a preset position of a gimbal camera according to claim 2, characterized in that, The specific steps of step S2 are as follows: Step S2.1: Use a feature point matching algorithm to match the clear image with the pre-stored standard image to obtain preliminary feature point pairs. Then filter out feature point pairs that do not meet the epipolar constraint to obtain the final image matching feature point pairs. Step S2.2: If the number of feature point pairs matched in the final image is less than the preset value, it is determined that the match is incorrect. At this time, the control module controls the gimbal camera to move in the horizontal and vertical directions in sequence to gradually adjust the gimbal pose. After adjustment, step 1 is executed again. Otherwise, the match is successful, that is, the scene captured by the clear image and the standard image is the same scene, and step S2.3 is executed. Step S2.3: Calculate the relative pose of the final image matching feature point pairs based on the clear image and the standard image, specifically as follows: Let the intrinsic parameter matrix K of the gimbal camera be: Among them, f u and f v Let f0 and v0 be the equivalent pixel focal lengths in the X and Y directions, respectively, and u0 and v0 be the X-axis and Y-axis coordinates of the principal pixel of the gimbal camera, respectively. Since the scaling factor in the horizontal and vertical directions is usually equal during the manufacturing process of a gimbal camera, i.e., f0... u =f v =s, where s is the overall scaling factor, and K is: Using the camera coordinate system of the standard image as the world coordinate system, let the direction vector of the projection from any point in the world coordinate system onto the gimbal camera coordinate system be P = [a, b, c]. T Let a, b, c represent the projections of the direction vector P onto the X, Y, and Z axes, respectively. (a, b, c) satisfy a 2 +b 2 +c 2 =1, a<<1, b<<1, c≈1, the coordinates (u, v) of this point projected onto the pixel plane are: After the gimbal camera rotates relative to the standard image by angles φ and θ around the X and Y axes respectively, the current image to be matched is obtained. The projection direction vector of the same point becomes P′. The transformation process is as follows: When photographing a distant target, if the target appears in both the standard image and the current image to be matched, then θ << 1 and φ << 1. In this case, sinθ ≈ θ, sinφ ≈ φ, cosθ ≈ 1, and cosφ ≈ 1. P′ is approximately: After ignoring the second-order minterms aθφ, bφ, and aθ: The coordinates (u′, v′) of P′ projected onto the pixel plane are: The pixel offset between the current image to be matched and the standard image is: At this point, only translation and scaling exist between the standard image and the current image to be matched. The homography matrix between the two images is equivalent to: The pixel offset is proportional to the rotation angle of the gimbal camera. By adjusting the angle of the gimbal camera, the pixel offset is made approximately equal to 0, so that the principal points of the two images coincide. The relative pose is calculated using a homography matrix-based method and the final image matching feature point pairs. Specifically: Let the sharp image be I1 and the standard image be I2. The coordinates of the final image matching feature point pairs obtained from the sharp image I1 and the standard image I2 are respectively... and n is the final number of feature point pairs matched in the image. Let the homography matrix H of the sharp image (I1) and the standard image (I2) under any pose be defined as: The mapping relationship between feature matching points of the clear image (I1) and the standard image (I2) is defined as follows: Where k is the scaling factor, and the final feature matching point pairs in the two images, I1 (sharp image) and I2 (standard image), have the following relationship with the homography matrix: Through N groups and The final feature matching point pairs are used to list 2N linear equations, and the elements in H are solved using the least squares method, where N > 8; Since at this point, the only differences between the clear image I1 and the standard image I2 are jitter and scaling, H 12 H 21 H 31 and H 32 The homography matrix H between the clear image I1 and the standard image I2 is approximately 0. ts : H≈H ts The homography matrix H obtained by solving the relationship between the final feature matching point pairs between the clear image I1 and the standard image I2 and the homography matrix satisfies: Among them, H 11 and H 22 The relative scaling factors in the X and Y directions between the sharp image I1 and the standard image I2 are typically taken as their average. This average is used to guide the gimbal camera in adjusting the focus. 13 and H 23 These are the pixel offsets in the X and Y directions between the clear image I1 and the standard image I2, respectively. These two values ​​are used to guide the movement of the gimbal camera and perform pose alignment. In this step, via H 13 and H 23 Guide the gimbal camera to move until the real-time camera pose coincides with the camera pose of the standard image, and after coincidence, satisfy H 13 ≈0, H 23 ≈0.

4. The method for calibrating a preset position of a gimbal camera according to claim 3, characterized in that, The specific steps of step S3 are as follows: Step 3.1: After pose calibration, when the pan-tilt camera's shooting position coincides with the standard image shooting position, there is no relative offset between the two shooting positions, only relative scaling. The homography matrix is ​​equivalent to: New images are acquired again, and feature point pair matching and homography matrix calculation are performed sequentially. The scaling factor is then obtained based on the homography matrix. Step 3.2: If the scaling factor is not 1, the control module of the gimbal camera adjusts the focal length according to the difference between the scaling factor and 1. After adjusting the focal length, the image is re-acquired and step 3.1 is executed again. Otherwise, the scaling factor s satisfies |s-1|<∈, where ∈ is the minimum threshold, and the final calibration is completed.

5. A system for calibrating a preset position of a gimbal camera, characterized in that, include: Image acquisition and sharpness assessment module: Based on the currently set focal length and pose of the gimbal camera, a gradient-based sharpness assessment algorithm is used to judge the sharpness of the original images of the target scene acquired by the gimbal camera in order to obtain a sharp image. Matching and pose calibration module: Match the clear image with the standard image by feature points. If the matching fails, control the gimbal camera to rotate and execute step S1 again. If the matching is successful, solve the homography matrix based on the matched feature points to guide the gimbal camera to perform pose calibration. After pose calibration, a new image is acquired, the scaling factor is calculated, and the focal length is adjusted according to the scaling factor to complete the final calibration. Control module: Controls the gimbal camera based on instructions from the image acquisition and sharpness assessment module and the matching and pose calibration module.

6. The system for calibrating a preset position of a gimbal camera according to claim 5, characterized in that, The specific implementation steps of the image acquisition and sharpness assessment module are as follows: Step S1.1: When the gimbal camera receives the image acquisition command, it acquires the original image of the target scene based on the currently set focal length and pose. Step S1.2: Convert the original image to a grayscale image, and calculate the gradient magnitude of each pixel in the grayscale image using the Sobel operator or the Prevet operator. For the pixel at coordinates (i, j) in the grayscale image, its pixel value is f. (i，j) Calculate its horizontal gradient G x (i, j) and the vertical gradient G y (i, j), and based on the directional gradient G x (i, j) and the vertical gradient G y The gradient magnitude of the pixel at coordinates (i, j) is calculated using the following formula: G x (i, j)=f ((i+1，j+1)) +2f ((i，j+1)) +f ((i-1，j+1)) -f ((i-1，j-1)) -2f ((i，j-1)) -f ((i+1，j-1)) G y (i, j)=f ((i-1，j-1)) +2f ((i-1，j)) +f ((i-1，j+1)) -f ((i+1，j-1)) -2f ((i+1，j)) -f ((i+1，j+1)) Mean and variance of gradient magnitudes based on grayscale images: Where M is the mean of the gradient magnitude of the grayscale image, V is the variance of the gradient magnitude of the grayscale image, and the variance is used as the sharpness score. The larger the variance, the sharper the original image. N represents the number of pixels in the grayscale image. Step S1.3: If the variance is lower than the preset clarity threshold, the original image is determined to be unclear. At this time, the control module of the gimbal camera sends a command to its focus adjustment mechanism to adjust the direction, and the original image is re-acquired and step S1.2 is executed again. Otherwise, a clear image is obtained.

7. The system for calibrating a preset position of a gimbal camera according to claim 6, characterized in that, The specific implementation steps of pose calibration in the matching and pose calibration module are as follows: Step S2.1: Use a feature point matching algorithm to match the clear image with the pre-stored standard image to obtain preliminary feature point pairs. Then filter out feature point pairs that do not meet the epipolar constraint to obtain the final image matching feature point pairs. Step S2.2: If the number of feature point pairs matched in the final image is less than the preset value, it is determined that the match is incorrect. At this time, the control module controls the gimbal camera to move in the horizontal and vertical directions in sequence to gradually adjust the gimbal pose. After adjustment, step 1 is executed again. Otherwise, the match is successful, that is, the scene captured by the clear image and the standard image is the same scene, and step S2.3 is executed. Step S2.3: Calculate the relative pose of the final image matching feature point pairs based on the clear image and the standard image, specifically as follows: Let the intrinsic parameter matrix K of the gimbal camera be: Among them, f u and f v Let f0 and v0 be the equivalent pixel focal lengths in the X and Y directions, respectively, and u0 and v0 be the X-axis and Y-axis coordinates of the principal pixel of the gimbal camera, respectively. Since the scaling factor in the horizontal and vertical directions is usually equal during the manufacturing process of a gimbal camera, i.e., f0... u =f v =s, where s is the overall scaling factor, and K is: Using the camera coordinate system of the standard image as the world coordinate system, let the direction vector of the projection from any point in the world coordinate system onto the gimbal camera coordinate system be P = [a, b, c]. T Let a, b, c represent the projections of the direction vector P onto the X, Y, and Z axes, respectively. (a, b, c) satisfy a 2 +b 2 +c 2 =1, a<<1, b<<1, c≈1, the coordinates (u, v) of this point projected onto the pixel plane are: After the gimbal camera rotates relative to the standard image by angles φ and θ around the X-axis and T-axis respectively, the current image to be matched is obtained. The projection direction vector of the same point becomes P′. The transformation process is as follows: When photographing a distant target, if the target appears simultaneously in both the standard image and the current image to be matched, then θ << 1 and φ << 1. In this case, sinθ ≈ θ, sinφ ≈ φ, cosθ ≈ 1, and Cosφ ≈ 1. P′ is approximately: After ignoring the second-order minterms aθφ, bφ, and aθ: The coordinates (u′, v′) of P′ projected onto the pixel plane are: The pixel offset between the current image to be matched and the standard image is: At this point, only translation and scaling exist between the standard image and the current image to be matched. The homography matrix between the two images is equivalent to: The pixel offset is proportional to the rotation angle of the gimbal camera. By adjusting the angle of the gimbal camera, the pixel offset is made approximately equal to 0, so that the principal points of the two images coincide. The relative pose is calculated using a homography matrix-based method and the final image matching feature point pairs. Specifically: Let the sharp image be I1 and the standard image be I2. The coordinates of the final image matching feature point pairs obtained from the sharp image I1 and the standard image I2 are respectively... and n is the final number of feature point pairs matched in the image. Let the homography matrix H of the sharp image (I1) and the standard image (I2) under any pose be defined as: The mapping relationship between feature matching points of the clear image (I1) and the standard image (I2) is defined as follows: Where k is the scaling factor, and the final feature matching point pairs in the two images, I1 (sharp image) and I2 (standard image), have the following relationship with the homography matrix: Through N groups and The final feature matching point pairs are used to list 2N linear equations, and the elements in H are solved using the least squares method, where N > 8; Since at this point, the only differences between the clear image I1 and the standard image I2 are jitter and scaling, H 12 H 21 H 31 and H 32 The homography matrix H between the clear image I1 and the standard image I2 is approximately 0. ts : H≈H ts The homography matrix H obtained by solving the relationship between the final feature matching point pairs between the clear image I1 and the standard image I2 and the homography matrix satisfies: Among them, H 11 and H 22 The relative scaling factors in the X and Y directions between the sharp image I1 and the standard image I2 are typically taken as their average. This average is used to guide the gimbal camera in adjusting the focus. 13 and H 23 These are the pixel offsets in the X and Y directions between the clear image I1 and the standard image I2, respectively. These two values ​​are used to guide the movement of the gimbal camera and perform pose alignment. In this step, via H 13 and H 23 Guide the gimbal camera to move until the real-time camera pose coincides with the camera pose of the standard image, and after coincidence, satisfy H 13 ≈0, H 23 ≈0.

8. The system for calibrating a preset position of a gimbal camera according to claim 7, characterized in that, The specific steps in the matching and pose calibration module, which involve acquiring a new image after pose calibration, calculating a scaling factor, and adjusting the focal length based on the scaling factor to complete the final calibration, are as follows: Step 3.1: After pose calibration, when the pan-tilt camera's shooting position coincides with the standard image shooting position, there is no relative offset between the two shooting positions, only relative scaling. The homography matrix is ​​equivalent to: New images are acquired again, and feature point pair matching and homography matrix calculation are performed sequentially. The scaling factor is then obtained based on the homography matrix. Step 3.2: If the scaling factor is not 1, the control module of the gimbal camera adjusts the focal length according to the difference between the scaling factor and 1. After adjusting the focal length, the image is re-acquired and step 3.1 is executed again. Otherwise, the scaling factor s satisfies |s-1|<∈, where ∈ is a minimum threshold, and the final calibration is completed.

Images