An automatic calibration method and system for a camera with both bullet and PTZ control

Abstract

This invention relates to the field of security monitoring technology and discloses an automatic calibration method and system for PTZ cameras. The method includes acquiring the target's three-dimensional coordinates, the PTZ optical center displacement vector, and the PTZ image sequence; extracting target feature points and tracing them using pyramid optical flow to obtain their pixel coordinates and changes within the PTZ's field of view; calculating the parallax compensation angle using the displacement vector, and adjusting the PTZ pointing vector using an affine transformation matrix if the parallax exceeds a threshold; subsequently extracting the PTZ rotation angle and removing distortion, and obtaining a preliminary transformation matrix after verification and fusion; obtaining a compensation factor through illumination analysis, and obtaining an optimized matrix by combining it with equipment displacement data to correct the matrix; finally, mapping the target coordinates to calculate the theoretical attitude angle, calculating the tracking error between the theoretical attitude angle and the actual attitude angle of the PTZ, and locking the automatic tracking mode if the error meets the standard, thus completing the high-precision tracking calibration of the PTZ camera linkage. This method can achieve high-precision automatic calibration of the PTZ camera linkage, improving tracking accuracy and stability.

Description

Technical Field

[0001] This invention relates to the field of security monitoring technology, and in particular to an automatic calibration method and system for PTZ cameras. Background Technology

[0002] Currently, in the field of security monitoring, the bullet-and-tube camera linkage system has become the core equipment for urban security, traffic management, and protection of important locations. By combining the wide-range monitoring of fixed bullet cameras with the high-definition detail tracking of PTZ cameras, it significantly improves the breadth and accuracy of monitoring coverage. Meanwhile, image recognition technology, as the core support for target positioning and trajectory tracking, plays a key role in the target association of the linkage system.

[0003] In one existing technology, the calibration of the PTZ camera linkage often adopts manual target alignment or image recognition matching mode based on a fixed target. That is, the PTZ camera is manually adjusted to align with the target identified by the PTZ camera image, or the pixel coordinate correspondence is established by using the image features of a preset target to establish the linkage relationship between the two cameras. However, the spatial geometric differences during installation and the dynamic interference factors during operation are not fully considered.

[0004] Existing technologies suffer from the problem of balancing calibration effectiveness and tracking stability. In actual deployments, the PTZ camera and PTZ camera are not coaxially mounted, resulting in spatial displacement vector and angular differences due to non-coincident optical centers. Current image recognition matching relies solely on simple two-dimensional pixel mapping, ignoring the resulting parallax effect. Furthermore, changes in ambient lighting, PTZ camera lens distortion, and minute mechanical displacements after long-term operation can cause the initially established correspondence to quickly become invalid. Recalibration is time-consuming, laborious, and inconsistent, ultimately leading to pointing errors during PTZ camera rotation and tracking, resulting in target loss or tracking drift, thus restricting the high-precision application of the PTZ camera linkage system. Summary of the Invention

[0005] This invention provides an automatic calibration method and system for a camera with integrated camera and PTZ camera, so as to achieve high-precision automatic calibration and accurate collaborative tracking of integrated camera and PTZ camera, and ensure the accuracy of target positioning and the stability of linkage tracking.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides an automatic calibration method for a PTZ camera, comprising:

[0007] Acquire the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun; Feature points of the target are extracted from the image sequence, and pyramid optical flow tracking analysis is performed on the feature points to construct discrete motion trajectories, thereby obtaining the pixel coordinates and pixel coordinate changes of the target within the gun's field of view. Based on the pixel coordinate changes and the displacement vector, the PTZ camera rotation parameters are calculated and the PTZ camera pointing vector is generated; The real-time rotation angle is extracted from the PTZ camera pointing vector and distortion removal is performed to obtain the corrected PTZ camera angle. The consistency verification and fusion processing of the PTZ camera angle and the pixel coordinates are performed to obtain the preliminary transformation matrix. The initial transformation matrix is ​​used to process subsequent image frames, extract the brightness histogram of the current frame and perform differential analysis to obtain illumination change feature quantities. If the illumination change feature quantities meet the preset conditions, the illumination change feature quantities are converted into illumination compensation factors. The preliminary transformation matrix is ​​corrected based on the illumination compensation factor and the real-time acquired equipment displacement data to obtain an optimized transformation matrix. The target's three-dimensional coordinates are mapped to the local coordinate system through the optimized transformation matrix, the theoretical attitude angle of the target is calculated, and the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera is calculated to obtain the tracking error. If the tracking error is lower than the preset error locking threshold, the automatic tracking mode is locked to obtain the calibration result of the high-precision tracking of the gun-ball linkage.

[0008] Preferably, acquiring the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun includes: The camera captures a continuous sequence of images of the monitored area, and the camera obtains the three-dimensional coordinates of the target within the monitored area and the point cloud data of the camera and PTZ camera shells through a three-dimensional coordinate measuring device. Based on the point cloud data of the outer shell, point cloud registration is performed in combination with the preset equipment design parameters to calculate the spatial coordinates of the optical center of the bullet camera and the optical center of the PTZ camera, and the vector difference between the spatial coordinates of the optical center of the bullet camera and the optical center of the PTZ camera is calculated to generate the initial displacement vector. The attitude quaternions of the camera and the PTZ camera are obtained. Based on a preset transformation formula, the attitude quaternions are converted into a relative rotation matrix. The initial displacement vector is rotated using the relative rotation matrix to obtain the displacement vector between the optical centers of the camera and the PTZ camera.

[0009] Preferably, the step of extracting feature points of the target from the image sequence, performing pyramid optical flow tracking analysis on the feature points, constructing discrete motion trajectories, and obtaining the pixel coordinates and pixel coordinate changes of the target within the gun's field of view includes: Pixel grayscale gradient analysis is performed on the first frame of the image sequence to identify pixels with corner features as initial target feature points; The initial target feature points are input into a preset pyramid optical flow tracking model to calculate the predicted position coordinates of the initial target feature points in the current frame image coordinate system. Based on the reverse optical flow tracking error of the predicted position coordinates, feature points with errors within a preset range are selected to obtain accurate feature points; The precise feature points are sorted according to the timestamps to construct a discrete motion trajectory point series. Polynomial curve fitting is performed on the motion trajectory point series to obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view.

[0010] Preferably, the step of calculating the PTZ camera rotation parameters and generating the PTZ camera pointing vector based on the pixel coordinate changes and the displacement vector includes: By combining the focal length of the PTZ camera with the position of the optical center, the pixel coordinate changes and the displacement vector are mapped into a spatial motion vector; Obtain the baseline distance between the bullet camera and the PTZ camera, and calculate the parallax compensation angle based on the geometric relationship between the spatial motion vector and the baseline distance; Compare the parallax compensation angle with the preset parallax judgment threshold. If it does not exceed the threshold, directly call the current preset PTZ camera rotation parameters as PTZ camera rotation data, and generate the PTZ camera pointing vector based on the PTZ camera rotation data. If the error exceeds the limit, the horizontal and vertical parallax compensation angles are calculated based on the horizontal and vertical components of the pixel coordinate changes, and the horizontal and vertical rotation angles of the PTZ camera are corrected to obtain the parallax-corrected PTZ camera rotation data. An adjusted PTZ camera pointing vector is then generated based on the PTZ camera rotation data.

[0011] Preferably, the step of extracting the real-time rotation angle from the PTZ camera pointing vector and performing distortion correction processing to obtain the corrected PTZ camera angle, and performing consistency verification and fusion processing on the PTZ camera angle and the pixel coordinates to obtain a preliminary transformation matrix, includes: The PTZ camera pointing vector is analyzed to extract the real-time rotation angle, which includes the horizontal azimuth angle and the vertical pitch angle; The real-time rotation angle is corrected by calling a preset PTZ camera lens distortion model to obtain the corrected PTZ camera angle. Based on the preset internal parameters of the camera, the displacement vector, and the baseline distance, the theoretical projection pixel coordinates of the PTZ camera angle on the camera image plane are calculated. Calculate the reprojection error value between the theoretical projected pixel coordinates and the pixel coordinates. If the reprojection error value meets the preset consistency constraint condition, it is determined that the two correspond. The corrected PTZ camera angle and the pixel coordinates are fused to generate a preliminary transformation matrix containing rotation and translation components. If the reprojection error value does not meet the preset consistency constraint, it is determined that the two do not correspond, the current feature point is discarded, and the process returns to the step of extracting the feature points of the target.

[0012] Preferably, the step of processing subsequent image frames using the preliminary transformation matrix, extracting the current frame's brightness histogram, and performing differential analysis to obtain illumination change feature quantities, and if the illumination change feature quantities meet preset conditions, then converting the illumination change feature quantities into illumination compensation factors, including: The initial transformation matrix is ​​used to perform spatial mapping processing on subsequent image frames to separate the brightness component of the image, and a grayscale distribution map is constructed based on the brightness component. Statistically analyze the percentage of pixels with different brightness values ​​in the grayscale distribution map to generate a brightness histogram for the current frame. Calculate the difference between the current frame brightness histogram and the preset reference frame brightness histogram, and accumulate the difference by weighting to obtain the global illumination change feature. The illumination change characteristic quantity is compared with the preset illumination change threshold. If the preset condition is met, the illumination change characteristic quantity is converted into an illumination compensation factor according to a preset ratio. If the condition is not met, the illumination compensation factor is set to a preset benchmark value.

[0013] Preferably, the step of correcting the preliminary transformation matrix based on the illumination compensation factor and the real-time acquired device displacement data to obtain an optimized transformation matrix includes: The illumination compensation factor is weighted and calculated with each component of the preliminary transformation matrix to generate an intermediate transformation matrix; Alignment analysis is performed between the real-time mechanical displacement data output by the displacement monitoring sensor and the spatial reference of the intermediate state transformation matrix to calculate the device displacement vector containing direction and amplitude. Compare the amplitude of the device displacement vector with the preset micro-motion detection threshold. If it does not exceed the threshold, it is determined that no displacement calibration is required, and the intermediate state transformation matrix is ​​directly used as the optimized transformation matrix. If the value exceeds the limit, the spatial position offset is calculated based on the device displacement vector. An adaptive parameter set containing horizontal offset correction values ​​and vertical offset correction values ​​is constructed. The adaptive parameter set is then superimposed on the intermediate state transformation matrix for a second iteration update to obtain the optimized transformation matrix.

[0014] Preferably, the step of mapping the target's three-dimensional coordinates to a local coordinate system through the optimized transformation matrix, calculating the target's theoretical attitude angle, and calculating the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera to obtain the tracking error includes: The optimized transformation matrix is ​​used to map the target's three-dimensional coordinates to the PTZ camera's local coordinate system; Based on the target position in the local coordinate system of the PTZ camera, the theoretical azimuth angle and theoretical pitch angle are calculated, which together form the theoretical attitude angle of the target; The PTZ camera receives actual attitude angle data fed back by its built-in sensors, and the actual attitude angle data includes the actual azimuth angle and the actual pitch angle. The difference between the actual azimuth angle and the theoretical azimuth angle, and the difference between the actual pitch angle and the theoretical pitch angle are calculated respectively. The two sets of differences are integrated into a tracking error signal, and the tracking error corresponding to the tracking error signal is extracted.

[0015] Preferably, if the tracking error is lower than a preset error locking threshold, then locking the automatic tracking mode to obtain the calibration result of the high-precision tracking of the gun-ball linkage includes: Compare the tracking error with a preset error lockout threshold. If the error is not lower than the error lockout threshold, return to re-execute the step of calculating the disparity compensation angle based on the pixel coordinate change and the displacement vector, and the subsequent steps. If the error is below the aforementioned error lockout threshold, a lockout signal for automatic tracking mode is generated, and the servo control parameters of the PTZ camera are fixed based on the lockout signal. Based on the servo control parameters, the PTZ camera is controlled to follow the target movement, and the calibration result of high-precision tracking with gun-ball linkage is obtained.

[0016] Secondly, the present invention provides an automatic calibration system for a camera with a PTZ (Panorama) linkage mechanism, comprising: The data acquisition module is used to acquire the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun. The feature tracking module is used to extract feature points of the target from the image sequence, perform pyramid optical flow tracking analysis on the feature points, construct discrete motion trajectories, and obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view. The parallax compensation module is used to calculate the PTZ camera rotation parameters and generate the PTZ camera pointing vector based on the pixel coordinate changes and the displacement vector; The matrix generation module is used to extract the real-time rotation angle from the PTZ camera pointing vector and perform distortion removal processing to obtain the corrected PTZ camera angle. The consistency verification and fusion processing of the PTZ camera angle and the pixel coordinates are performed to obtain a preliminary transformation matrix. The illumination compensation module is used to process subsequent image frames using the preliminary transformation matrix, extract the brightness histogram of the current frame and perform differential analysis to obtain illumination change feature quantities. If the illumination change feature quantities meet the preset conditions, the illumination change feature quantities are converted into illumination compensation factors. The matrix optimization module is used to correct the preliminary transformation matrix based on the illumination compensation factor and the real-time acquired equipment displacement data to obtain an optimized transformation matrix; The error calculation module is used to map the three-dimensional coordinates of the target to the local coordinate system through the optimized transformation matrix, calculate the theoretical attitude angle of the target, calculate the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera, and obtain the tracking error. The tracking lock module is used to lock the automatic tracking mode if the tracking error is lower than a preset error lock threshold, so as to obtain the calibration result of the high-precision tracking of the gun-ball linkage.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention calculates the precise displacement vector of the optical center of the two cameras, extracts the target feature points and tracks the pixel coordinate change trend, calculates the parallax compensation angle in combination with parameters and adjusts the pointing of the camera as needed through the affine transformation matrix, breaks through the limitation of ignoring the optical center displacement and parallax effect in the traditional method, effectively compensates for non-coaxial installation deviation, and improves the target pointing accuracy of the camera.

[0018] (2) This invention performs distortion correction on the rotation angle of the PTZ camera and generates a preliminary coordinate transformation matrix after consistency verification. It obtains the illumination compensation factor by combining the difference in brightness histogram, performs adaptive calibration based on mechanical displacement data, and continuously optimizes the matrix to solve the problem of failure of traditional calibration due to multi-source interference, avoid the accumulation of tracking deviation, and improve linkage stability.

[0019] (3) The present invention obtains the theoretical attitude angle by mapping the target through the optimized transformation matrix, extracts the tracking error by comparing the actual angle of the PTZ camera, locks the automatic tracking mode and solidifies the servo parameters under the threshold, replaces the manual calibration method, solves the tracking drift and target loss problems, and improves the efficiency of linkage emergency response and positioning accuracy. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart of an automatic calibration method for a PTZ camera provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of an automatic calibration method system for a PTZ camera provided in the second embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figure 1 The first embodiment of the present invention provides an automatic calibration method for a camera with a PTZ sensor, comprising the following steps: S11, acquire the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun; S12, extract the feature points of the target from the image sequence, perform pyramid optical flow tracking analysis on the feature points, construct discrete motion trajectories, and obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view; S13, Calculate the PTZ camera rotation parameters and generate the PTZ camera pointing vector based on the pixel coordinate changes and the displacement vector; S14, extract the real-time rotation angle from the PTZ camera pointing vector and perform distortion removal processing to obtain the corrected PTZ camera angle. Perform consistency verification and fusion processing on the PTZ camera angle and the pixel coordinates to obtain the preliminary transformation matrix. S15, the initial transformation matrix is ​​used to process subsequent image frames, the brightness histogram of the current frame is extracted and differential analysis is performed to obtain the illumination change feature quantity. If the illumination change feature quantity meets the preset conditions, the illumination change feature quantity is converted into an illumination compensation factor. S16, Based on the illumination compensation factor and the real-time acquired equipment displacement data, the preliminary transformation matrix is ​​corrected to obtain the optimized transformation matrix; S17, map the target's three-dimensional coordinates to the local coordinate system through the optimized transformation matrix, calculate the target's theoretical attitude angle, calculate the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera, and obtain the tracking error; S18, if the tracking error is lower than the preset error locking threshold, then lock the automatic tracking mode to obtain the calibration result of the high-precision tracking of the gun-ball linkage.

[0023] In step S11, the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun are acquired, including: The camera captures a continuous sequence of images of the monitored area, and the camera obtains the three-dimensional coordinates of the target within the monitored area and the point cloud data of the camera and PTZ camera shells through a three-dimensional coordinate measuring device. Based on the point cloud data of the outer shell, point cloud registration is performed in combination with the preset equipment design parameters to calculate the spatial coordinates of the optical center of the bullet camera and the optical center of the PTZ camera, and the vector difference between the spatial coordinates of the optical center of the bullet camera and the optical center of the PTZ camera is calculated to generate the initial displacement vector. The attitude quaternions of the camera and the PTZ camera are obtained. Based on a preset transformation formula, the attitude quaternions are converted into a relative rotation matrix. The initial displacement vector is rotated using the relative rotation matrix to obtain the displacement vector between the optical centers of the camera and the PTZ camera.

[0024] It should be noted that when the bullet camera acquires continuous image sequences of the monitored area, the frame rate is set to 25fps according to the general standard in the security monitoring industry. The image resolution is 1920×1080 to match the actual monitoring scenario requirements of the bullet camera. The acquisition process ensures uniform time intervals between frames and no significant target loss to meet the inter-frame correlation requirements for subsequent feature point tracking. The 3D coordinate measurement equipment uses a high-precision laser scanner specifically designed for the security field, with a measurement accuracy controlled at ±0.1mm. This equipment simultaneously completes two data acquisition tasks: the 3D spatial coordinates of the target within the monitored area and the point cloud data of the bullet and PTZ cameras' casings. The point cloud data acquisition density is no less than 100 points / square centimeter to ensure complete extraction of the geometric feature points of the equipment casing, providing sufficient data support for subsequent optical center coordinate calculation.

[0025] It is worth noting that the point cloud registration operation is based on the original equipment design parameters of the bullet camera and PTZ camera. These design parameters include calibration parameters such as the geometric dimensions of the casing and the relative position coordinates of the optical center within the casing. The registration process employs an iterative nearest-point algorithm to register the scanned casing point cloud data with the standard point cloud model in the design parameters. The registration error is controlled within ±0.2mm to ensure consistency between the point cloud data and the actual equipment structure. After registration, based on the position parameters of the optical center relative to the casing, the spatial coordinates of the bullet camera's optical center and the PTZ camera's optical center are calculated. An initial displacement vector is generated through spatial vector difference calculation, achieving quantitative calculation of the initial displacement vector. For example, in a road monitoring scenario, the calculated spatial coordinates of the bullet camera's optical center are (1.5, 2.0, 3.0) m, and the PTZ camera's optical center are (2.0, 2.5, 3.5) m. The initial displacement vector can then be calculated. =(0.5,0.5,0.5)m.

[0026] It is worth further explaining that the attitude quaternions of the camera module and the PTZ camera are obtained by a combination of a gyroscope and an accelerometer attitude sensor built into the device. The sensor sampling frequency is 100Hz, and the output attitude quaternion form is as follows: Where w is the real part of the quaternion, and i, j, and k are the imaginary parts of the quaternion. The transformation from attitude quaternion to relative rotation matrix adopts a common transformation formula in the security field.

[0027] The quaternion is solved into a 3×3 relative rotation matrix R using this formula. The initial displacement vector is then rotated using the relative rotation matrix R. Through matrix-space vector multiplication, the final displacement vector between the optical centers of the gun and PTZ cameras is obtained. This vector serves as the basis for subsequent parallax compensation calculations. This rotation transformation process can reduce the displacement calculation error caused by equipment attitude differences to within ±0.05m, effectively improving the accuracy of displacement vector calculation. It should be noted that the commas in the above matrix are only used to isolate matrix elements and have no other meaning.

[0028] In another alternative implementation, if the monitoring scenario has limited installation space for the equipment and the laser scanner cannot collect point cloud data at close range, a visual calibration method can be used to replace laser scanning to obtain the shell feature point data of bullet and PTZ cameras. Combined with the equipment design parameters, the optical center spatial coordinates can be calculated. The attitude quaternion can also be obtained through an external visual positioning device, fused with the data from the device's built-in sensors, and then converted into a relative rotation matrix R to further improve the accuracy of the attitude data and ensure that the displacement vector calculation results are adapted to different field deployment environments.

[0029] In step S12, feature points of the target are extracted from the image sequence, pyramid optical flow tracking analysis is performed on the feature points, discrete motion trajectories are constructed, and the pixel coordinates and pixel coordinate changes of the target within the gun's field of view are obtained, including: Pixel grayscale gradient analysis is performed on the first frame of the image sequence to identify pixels with corner features as initial target feature points; The initial target feature points are input into a preset pyramid optical flow tracking model to calculate the predicted position coordinates of the initial target feature points in the current frame image coordinate system. Based on the reverse optical flow tracking error of the predicted position coordinates, feature points with errors within a preset range are selected to obtain accurate feature points; The precise feature points are sorted according to the timestamps to construct a discrete motion trajectory point series. Polynomial curve fitting is performed on the motion trajectory point series to obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view.

[0030] It should be noted that when performing pixel grayscale gradient analysis on the first frame of the image sequence, the Sobel operator is used to calculate the grayscale gradient values ​​of the pixels in the x and y directions respectively. The comprehensive gradient magnitude of the pixels is calculated using the gradient magnitude formula. Corner features are determined based on the grayscale change rate of neighboring pixels in the image. After verification with multiple batches of on-site monitoring images, pixels with gradient magnitudes greater than 20 have significant corner features and meet the requirements for target feature point recognition. These pixels are marked as initial target feature points. This threshold is set based on the grayscale distribution characteristics of bullet camera images in the security monitoring field and the general industrial standards for corner detection. After inputting the initial target feature points into the preset pyramid optical flow tracking model, the model first constructs a 4-layer multi-scale image pyramid through Gaussian downsampling. The number of layers is based on the optical flow tracking equipment manual and industry-standard settings. The Lucas-Kanade algorithm is used to solve the basic optical flow equations for the feature points in each layer of the image. The influence of image scale changes on the tracking results is eliminated through multi-scale iterative calculation. Finally, the predicted position coordinates of the initial target feature points in the current frame image coordinate system are obtained.

[0031] It is worth noting that when selecting precise feature points based on the reverse optical flow tracking error of the predicted position coordinates, the predicted position coordinates of the current frame are traced back to the first frame of the image sequence, and the pixel deviation between the traced position and the initial target feature point position is calculated. This pixel deviation is the reverse optical flow tracking error. The preset error range is set to pixel deviation ≤ 3 based on historical monitoring data statistics and multiple experimental verifications. This threshold is comprehensively set in combination with the pixel matching accuracy requirements of the gun-ball linkage, the noise level of the security monitoring image, and the pixel offset caused by changes in illumination. The deviation range of 3 pixels is the fault tolerance range for feature point tracking. Feature points within this range are retained as precise feature points, and feature points outside the range are removed to eliminate tracking errors caused by changes in illumination and image noise. When sorting precise feature points based on timestamps, the coordinate information of each precise feature point is arranged in ascending order according to the timestamp of the gun image acquisition, forming a discrete motion trajectory point series. A quadratic polynomial is used to fit the trajectory point series. During the fitting process, the polynomial coefficients are solved using the least squares method to minimize the sum of squares of the pixel deviations between the fitted curve and the trajectory point series. After fitting calculation, the pixel coordinates of the target in the gun's field of view can be directly obtained. By calculating the difference between the pixel coordinates at adjacent times, the change of the target's pixel coordinates in the gun's field of view is obtained. This fitting method can effectively smooth out the random errors of discrete trajectory points and accurately reflect the actual motion trend of the target.

[0032] In step S13, the PTZ camera rotation parameters are calculated based on the pixel coordinate changes and the displacement vector, and a PTZ camera pointing vector is generated, including: By combining the focal length of the PTZ camera with the position of the optical center, the pixel coordinate changes and the displacement vector are mapped into a spatial motion vector; Obtain the baseline distance between the bullet camera and the PTZ camera, and calculate the parallax compensation angle based on the geometric relationship between the spatial motion vector and the baseline distance; Compare the parallax compensation angle with the preset parallax judgment threshold. If it does not exceed the threshold, directly call the current preset PTZ camera rotation parameters as PTZ camera rotation data, and generate the PTZ camera pointing vector based on the PTZ camera rotation data. If the error exceeds the limit, the horizontal and vertical parallax compensation angles are calculated based on the horizontal and vertical components of the pixel coordinate changes, and the horizontal and vertical rotation angles of the PTZ camera are corrected to obtain the parallax-corrected PTZ camera rotation data. An adjusted PTZ camera pointing vector is then generated based on the PTZ camera rotation data.

[0033] It should be noted that when mapping the spatial motion vector between the PTZ camera's focal length and optical center position, the focal length *f* calibrated in the PTZ camera's manual and the optical center pixel coordinates obtained from factory calibration are first retrieved. Simultaneously, the pixel pitch *d* of the bullet camera's imaging chip is extracted. The change in pixel coordinates within the bullet camera's field of view is multiplied by the pixel pitch to obtain the physical displacement of the imaging plane. Then, combined with the optical center displacement vectors of the bullet and PTZ cameras obtained from the three-dimensional coordinate measurement equipment in step S11, the pixel domain is transformed to the spatial domain using the three-dimensional spatial coordinate transformation formula, resulting in the spatial motion vector *V*. Based on the previously obtained optical center coordinates of the bullet and PTZ cameras, the baseline distance between the two cameras is calculated using the spatial straight-line distance formula. Based on the principle of trigonometric parallax, the magnitude of the spatial motion vector is first solved. Then, the disparity compensation angle is calculated using the arctangent function. .

[0034] It is worth noting that the parallax judgment threshold was determined through extensive on-site monitoring experiments. These experiments covered monitoring distances ranging from 50 to 500 meters, including typical monitoring targets such as pedestrians and vehicles. Combined with the ±0.1° rotation accuracy of the PTZ camera servo system and the high-precision requirement of a pixel deviation ≤3 pixels during PTZ camera-PTZ linkage, the parallax judgment threshold was determined to be 0.5°. This threshold was set because when the parallax compensation angle exceeds 0.5°, the pointing pixel deviation of the PTZ camera tracking the target will exceed 3 pixels, failing to meet the linkage tracking accuracy requirements. The actual calculated value of the parallax compensation angle... A direct numerical comparison is made with 0.5°, and the comparison process uses a floating-point number comparison bit by bit to ensure the accuracy of the judgment result, providing a clear and quantifiable basis for the subsequent differential processing of the PTZ camera rotation parameters.

[0035] At the parallax compensation angle In the comparison with the preset threshold, if For angles ≤0.5°, the horizontal rotation angle α and vertical rotation angle β of the PTZ camera, preset based on the target positioning result of the bullet camera, are directly retrieved as the PTZ camera rotation data. According to the spatial coordinate system transformation rules, the rotation data is mapped to the PTZ camera pointing vector. .like The horizontal variation of the target pixel coordinates within the gun's field of view Vertical variation Combining the PTZ camera focal length f, pixel pitch d, PTZ baseline distance D, and spatial motion vector model The horizontal parallax compensation angle is calculated using geometric projection relationships. With vertical parallax compensation angle The calculation formula is:

[0036]

[0037] The calculated horizontal and vertical parallax compensation angles are directly superimposed onto the preset horizontal and vertical rotation angles of the PTZ camera to obtain the corrected horizontal rotation angle. Vertical rotation angle This angle data is the PTZ camera rotation data after parallax correction. Then, according to the spatial coordinate system transformation formula, [the data is then processed]. Mapped to a three-dimensional unit vector, generating the adjusted PTZ camera pointing vector. In step S14, the real-time rotation angle is extracted from the PTZ camera pointing vector and distortion correction is performed to obtain the corrected PTZ camera angle. The consistency verification and fusion processing of the PTZ camera angle and the pixel coordinates are then performed to obtain a preliminary transformation matrix, including: The PTZ camera pointing vector is analyzed to extract the real-time rotation angle, which includes the horizontal azimuth angle and the vertical pitch angle; The real-time rotation angle is corrected by calling a preset PTZ camera lens distortion model to obtain the corrected PTZ camera angle. Based on the preset internal parameters of the camera, the displacement vector, and the baseline distance, the theoretical projection pixel coordinates of the PTZ camera angle on the camera image plane are calculated. Calculate the reprojection error value between the theoretical projected pixel coordinates and the pixel coordinates. If the reprojection error value meets the preset consistency constraint condition, it is determined that the two correspond. The corrected PTZ camera angle and the pixel coordinates are fused to generate a preliminary transformation matrix containing rotation and translation components. If the reprojection error value does not meet the preset consistency constraint, it is determined that the two do not correspond, the current feature point is discarded, and the process returns to the step of extracting the feature points of the target.

[0038] It should be noted that when extracting the real-time rotation angle from the PTZ camera's pointing vector, the PTZ camera's pointing vector is a three-dimensional unit vector. Based on the general transformation rules from the spatial rectangular coordinate system to the spherical coordinate system, the horizontal azimuth angle α and the vertical pitch angle β are calculated using the following formulas.

[0039]

[0040] in, The pointing angle of the PTZ camera on the horizontal plane, range arrive , This refers to the pointing angle of the PTZ camera in the vertical plane, ranging from -90° to 90°. Horizontal azimuth angle. and vertical pitch angle Together, they constitute the real-time rotation angle. When using a preset PTZ camera lens distortion model for distortion correction, this model incorporates the radial distortion coefficient calibrated by a target test before the device leaves the factory. , , With tangential distortion coefficient , First, map the real-time rotation angle to the pixel coordinates (x, y) of the PTZ camera's imaging plane, and then calculate the Euclidean distance from the pixel to the optical center of the PTZ camera's imaging plane before distortion correction. Through distortion correction formula

[0041]

[0042]

[0043] in, The corrected distance from the pixel to the optical center. These are the corrected horizontal pixel coordinates. The corrected vertical pixel coordinates are then mapped to angle values ​​to obtain the corrected PTZ camera angle. This process eliminates angle deviations caused by lens barrel and pincushion distortions. For example, if the original real-time rotation angles are 87.3° and 12.6°, they will be corrected to 87.1° and 12.8°.

[0044] It is worth noting that when calculating the theoretical projected pixel coordinates by combining the preset internal parameters of the camera, the optical center displacement vectors of the camera and PTZ, and the baseline distance, the internal parameters of the camera are the factory-calibrated focal length f and optical center pixel coordinates. Both the pixel pitch d and the target angle are pre-stored in the system. First, based on the corrected PTZ camera angle, the optical center displacement vector, and the baseline distance, the three-dimensional spatial coordinates (X, Y, Z) of the target relative to the optical center of the PTZ camera are calculated using spatial geometric relationships. Then, based on the pinhole camera imaging model, the formula is used to...

[0045]

[0046] The theoretical projected pixel coordinates were calculated. When calculating the reprojection error value, the pixel coordinates actually detected by the camera are used. Based on this, the reprojection error value *e* is calculated using the Euclidean distance formula. The threshold for the consistency constraint, verified through extensive field experiments covering different monitoring scenarios, is set at 3 pixels. This threshold is determined by combining the pixel accuracy of the bullet camera's 1920×1080 resolution, the residual error after PTZ lens distortion correction, and the pixel deviation in spatial coordinate mapping. An error range of 3 pixels effectively determines the spatial correspondence between the PTZ camera angle and the bullet camera's pixel coordinates. If the error exceeds this range, it indicates a significant deviation in the mapping relationship between the two cameras. For example, if the theoretical projected pixel coordinates are (312, 428) and the actual observed coordinates are (314, 430), the calculated reprojection error value is approximately 2.8 pixels, satisfying the consistency constraint.

[0047] It is worth further explaining that when the reprojection error value satisfies e≤3 pixels, the PTZ camera angle and the bullet camera pixel coordinates are determined to correspond to each other, and the corrected PTZ camera angle is then used. , Convert to rotation matrix

[0048] Using the optical center displacement vectors of the two machines (gun and ball) as the translation component t, a homogeneous preliminary transformation matrix containing rotation and translation components is generated. When the reprojection error value does not meet the consistency constraint, it is determined that the two cannot form a valid correspondence. The current target feature point is discarded directly, and the step of extracting the target feature point is returned to start the feature point tracking and subsequent full-process calculation again. This discarding mechanism can effectively eliminate invalid data caused by incomplete distortion correction and feature point tracking error. The commas in this matrix are also only used to separate matrix elements.

[0049] In step S15, the preliminary transformation matrix is ​​used to process subsequent image frames, the brightness histogram of the current frame is extracted, and differential analysis is performed to obtain illumination change feature quantities. If the illumination change feature quantities meet preset conditions, the illumination change feature quantities are converted into illumination compensation factors, including: The initial transformation matrix is ​​used to perform spatial mapping processing on subsequent image frames to separate the brightness component of the image, and a grayscale distribution map is constructed based on the brightness component. Statistically analyze the percentage of pixels with different brightness values ​​in the grayscale distribution map to generate a brightness histogram for the current frame. Calculate the difference between the current frame brightness histogram and the preset reference frame brightness histogram, and accumulate the difference by weighting to obtain the global illumination change feature. The illumination change characteristic quantity is compared with the preset illumination change threshold. If the preset condition is met, the illumination change characteristic quantity is converted into an illumination compensation factor according to a preset ratio. If the condition is not met, the illumination compensation factor is set to a preset benchmark value.

[0050] It should be noted that when using the preliminary transformation matrix to perform spatial mapping processing on subsequent image frames, matrix operations are used to map the image coordinates under the PTZ camera's field of view to the bullet camera's image coordinate system, simultaneously separating the image's luminance component. A grayscale distribution map is then constructed based on this luminance component. This distribution map reflects the brightness characteristics of different areas of the image, providing fundamental data for illumination change analysis. For example, in a certain monitoring scene, the grayscale distribution map shows that the luminance values ​​on the left side of the screen are concentrated in the 80-120 range, while those on the right side are concentrated in the 150-180 range, exhibiting obvious uneven illumination. When calculating the pixel proportion of different luminance values ​​in the grayscale distribution map, the range is divided into luminance values ​​of 0-255 (0 being pure black and 255 being pure white). The proportion of pixels in each range to the total number of pixels is calculated, generating a luminance histogram for the current frame. The horizontal axis of the histogram represents the luminance value, and the vertical axis represents the pixel proportion.

[0051] It is worth noting that when calculating the difference between the current frame's brightness histogram and the reference frame's brightness histogram, the reference frame is an image frame acquired during the initial deployment phase with no lighting interference and a stable target state. The Bach distance algorithm is used to calculate the similarity between the two histograms, with a similarity range of 0-1. The closer to 1, the more similar the two histograms are, and the smaller the lighting change. The difference between 1 and the similarity S is then calculated to obtain the global lighting change feature quantity L, with L ranging from 0-1. The closer to 1, the more significant the lighting change. The lighting change threshold was set to 0.3 after verification through multiple experiments under different lighting scenarios. If the lighting change feature quantity L > 0.3, it is determined that the preset condition is met, and the value is adjusted according to a preset ratio. Convert to an illumination compensation factor k. For example, when the illumination change characteristic is 0.4, the compensation factor is 0.6. If L≤0.3, the illumination change is considered negligible, and the illumination compensation factor is set to the baseline value of 1.0 to ensure that the matrix is ​​not affected by invalid illumination fluctuations.

[0052] It is worth further explaining that the conversion ratio of the illumination compensation factor is set according to the general standard for image brightness adjustment in the security monitoring field. This ratio can effectively offset the impact of illumination changes on pixel coordinate mapping. For example, in a cloudy scene, the illumination change feature is 0.5, and the converted compensation factor is 0.5. After being applied to matrix adjustment, the deviation of the mapped pixel coordinates can be reduced to within 1 pixel, ensuring the consistency of the gun-ball linkage data.

[0053] In step S16, the preliminary transformation matrix is ​​corrected based on the illumination compensation factor and the real-time acquired equipment displacement data to obtain an optimized transformation matrix, including: The illumination compensation factor is weighted and calculated with each component of the preliminary transformation matrix to generate an intermediate transformation matrix; Alignment analysis is performed between the real-time mechanical displacement data output by the displacement monitoring sensor and the spatial reference of the intermediate state transformation matrix to calculate the device displacement vector containing direction and amplitude. Compare the amplitude of the device displacement vector with the preset micro-motion detection threshold. If it does not exceed the threshold, it is determined that no displacement calibration is required, and the intermediate state transformation matrix is ​​directly used as the optimized transformation matrix. If the value exceeds the limit, the spatial position offset is calculated based on the device displacement vector. An adaptive parameter set containing horizontal offset correction values ​​and vertical offset correction values ​​is constructed. The adaptive parameter set is then superimposed on the intermediate state transformation matrix for a second iteration update to obtain the optimized transformation matrix.

[0054] It should be noted that when weighting the illumination compensation factor with each component of the initial transformation matrix, element-wise multiplication is used. The illumination compensation factor is multiplied by the rotation and translation components of the matrix respectively to generate the intermediate transformation matrix. This operation allows each component of the matrix to adapt to the current illumination conditions. For example, when the compensation factor is 0.8, each element of the rotation component in the matrix is ​​multiplied by 0.8, achieving reverse correction of illumination deviation. A high-precision micro-displacement sensor is selected for displacement monitoring, with a measurement accuracy of ±0.01mm and a sampling frequency of 50Hz. It outputs the mechanical displacement of the device in real time along the horizontal x-axis, vertical y-axis, and depth z-axis. When aligning this data with the spatial reference of the intermediate transformation matrix, the device's initial spatial position is used as the reference to calculate the device displacement vector, which includes direction and amplitude. .

[0055] It is worth noting that the micro-motion detection threshold was set to 0.2mm based on long-term operational experiments. This threshold was set according to the minimum mechanical response accuracy of the PTZ camera servo system (0.01°) and the spatial mapping accuracy of the PTZ linkage (±0.05m), when the amplitude of the displacement vector... When the deviation is ≤0.2mm, the deviation of the PTZ camera rotation angle is less than 0.01°, which does not exceed the minimum response accuracy of the PTZ camera servo system. Therefore, the influence of equipment displacement on the linkage accuracy is negligible, and the intermediate state conversion matrix is ​​directly used as the optimized conversion matrix. If the amplitude of the displacement vector... When the displacement is greater than 0.2mm, the spatial position offset is calculated based on the displacement vector, and an adaptive parameter set containing horizontal offset correction values ​​and vertical offset correction values ​​is constructed. The calculation of the correction value is based on the geometric relationship between the displacement vector and the matrix space reference. For example, when the horizontal displacement is 0.3mm, the correction value is the pixel offset corresponding to 0.3mm.

[0056] It is worth further explaining that when the adaptive parameter set is superimposed on the intermediate transformation matrix for a second iteration update, the gradient descent method is used for three iterations, with a step size of 0.3 for each iteration, ensuring stable convergence of the matrix update process and obtaining the optimized transformation matrix. This second iteration update method can reduce the mapping error caused by device displacement to within 0.05mm, significantly improving the spatial mapping accuracy of the matrix.

[0057] In another optional implementation, if the monitored scene experiences strong vibration interference, the sampling frequency of the displacement sensor can be increased to 100Hz, while the micro-motion detection threshold can be adjusted to 0.1mm, and the iteration update step size can be shortened to 0.2, enhancing the matrix's adaptability to high-frequency micro-motion. Alternatively, a Kalman filter algorithm can be introduced to smooth the displacement data, reducing the impact of vibration noise on displacement vector calculation and further improving the stability of the optimized transformation matrix.

[0058] In step S17, the target's three-dimensional coordinates are mapped to the local coordinate system through the optimized transformation matrix, the theoretical attitude angle of the target is calculated, and the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera is calculated to obtain the tracking error, including: The optimized transformation matrix is ​​used to map the target's three-dimensional coordinates to the PTZ camera's local coordinate system; Based on the target position in the local coordinate system of the PTZ camera, the theoretical azimuth angle and theoretical pitch angle are calculated, which together form the theoretical attitude angle of the target; The PTZ camera receives actual attitude angle data fed back by its built-in sensors, and the actual attitude angle data includes the actual azimuth angle and the actual pitch angle. The difference between the actual azimuth angle and the theoretical azimuth angle, and the difference between the actual pitch angle and the theoretical pitch angle are calculated respectively. The two sets of differences are integrated into a tracking error signal, and the tracking error corresponding to the tracking error signal is extracted.

[0059] It should be noted that when using the optimized transformation matrix to map the target's 3D coordinates to the PTZ camera's local coordinate system, the PTZ camera's local coordinate system has the PTZ camera's optical center as its origin, the horizontal rightward direction as the x-axis, the vertical upward direction as the y-axis, and the lens pointing direction as the z-axis. The target's 3D coordinates in the global coordinate system are then calculated using matrix multiplication. Convert to local coordinate system This transformation process follows general industry standards for spatial coordinate system transformation, and the transformation error can be controlled within 0.03m. When calculating the theoretical attitude angle based on the target position and spatial geometric relationships in the local coordinate system, the following formula is used:

[0060]

[0061] in, The horizontal azimuth angle of the PTZ camera pointing at the target. The vertical pitch angle of the PTZ camera pointing at the target together constitutes the target's theoretical attitude angle. For example, if the target coordinates are (5,2,10)m in the local coordinate system and the PTZ camera is installed at a height of 3m, the calculated theoretical azimuth angle is approximately 26.56° and the theoretical pitch angle is approximately -5.71°.

[0062] It is worth noting that the PTZ camera uses a built-in gyroscope and tilt sensor to feed back the actual attitude angle data. The sensor's angle measurement accuracy is ±0.02°, and the output actual attitude angle data includes the actual azimuth angle. and actual pitch angle The data update frequency is kept consistent with the image acquisition frame rate at 25Hz to ensure time synchronization between angle data and image data. The difference Δα between the actual azimuth and the theoretical azimuth, and the difference Δβ between the actual pitch and the theoretical pitch are calculated separately. The square root of the sum of the squares of the two sets of differences is integrated into a tracking error signal. The tracking error corresponding to this signal is extracted, and the error value directly reflects the degree of deviation between the actual tracking attitude of the PTZ camera and the ideal attitude.

[0063] It is worth further explaining that the method for calculating the tracking error has been verified through multiple batches of gun-ball linkage tracking experiments, and can accurately quantify the tracking deviation. For example, with a theoretical azimuth angle of 26.56° and an actual azimuth angle of 26.60°, a theoretical elevation angle of -5.71° and an actual elevation angle of -5.68°, the calculated tracking error is approximately 0.05°, which meets the requirements for high-precision tracking.

[0064] In step S18, if the tracking error is lower than a preset error locking threshold, the automatic tracking mode is locked to obtain the calibration result of the high-precision tracking of the gun-ball linkage, including: Compare the tracking error with a preset error lockout threshold. If the error is not lower than the error lockout threshold, return to re-execute the step of calculating the disparity compensation angle based on the pixel coordinate change and the displacement vector, and the subsequent steps. If the error is below the aforementioned error lockout threshold, a lockout signal for automatic tracking mode is generated, and the servo control parameters of the PTZ camera are fixed based on the lockout signal. Based on the servo control parameters, the PTZ camera is controlled to follow the target movement, and the calibration result of high-precision tracking with gun-ball linkage is obtained.

[0065] It should be noted that the error lock-in threshold was set at 0.2° after extensive field tracking experiments. This threshold was set according to the industry standard for high-precision tracking of the PTZ camera. When the tracking error E ≥ 0.2°, the tracking accuracy of the PTZ camera is deemed to be substandard. The process of calculating the disparity compensation angle based on pixel coordinate changes and displacement vectors, as well as subsequent steps, is then repeated. The disparity compensation and matrix optimization processes are carried out again until the tracking error meets the requirements. If E < 0.2°, a lock-in signal for the automatic tracking mode is generated. This signal is a high-level logic signal that triggers the parameter solidification process of the PTZ camera servo system.

[0066] It is worth noting that when the servo control parameters of the PTZ camera are fixed based on the locking signal, the fixed parameters include motor speed, angle adjustment step size, and PID control coefficients. The motor speed is set to 5° / s, the angle adjustment step size is 0.01°, and the PID control coefficients are obtained through system self-tuning algorithm and optimization of the PTZ camera's mechanical characteristic parameters. After calculating the initial PID coefficients by fitting the transfer function to the PTZ camera's motor dynamic response data, iterative fine-tuning is performed through on-site PTZ camera tracking experiments to ensure that the PTZ camera's tracking response meets the requirements of no overshoot and no steady-state error. For example, the proportional coefficient Kp=2.5, integral coefficient Ki=0.1, and derivative coefficient Kd=0.5. These parameters are set based on the PTZ camera's mechanical performance parameters and on-site tracking requirements to ensure smooth tracking without overshoot.

[0067] It is worth further explaining that when the PTZ camera follows the target movement based on the servo control parameters, the camera adjusts its own attitude according to the real-time changes in the target position mapped by the optimized transformation matrix, so that the tracking error remains stable within 0.2°, resulting in a calibration result for high-precision tracking with PTZ camera linkage. This calibration result includes data such as the optimized coordinate transformation matrix, servo control parameters, and tracking error range, which can be stored locally on the system as reference parameters for subsequent tracking.

[0068] In another alternative implementation, for monitoring scenarios involving high-speed moving targets, the error lockout threshold can be adjusted to 0.1°, while the motor speed is increased to 10° / s, and the PID control coefficients are optimized to improve system response speed. For monitoring scenarios involving static or slowly moving targets, the error lockout threshold can be adjusted to 0.3°, and the motor speed reduced to 2° / s to reduce equipment energy consumption and adapt to the tracking needs of different types of targets.

[0069] In summary, this invention discloses an automatic calibration method for a bullet-and-tube camera, comprising: acquiring the target's three-dimensional coordinates, the displacement vector between the optical centers of the bullet and the PTZ cameras, and an image sequence of the bullet camera; extracting feature points of the target from the image sequence, performing pyramid optical flow tracking analysis on the feature points, constructing a discrete motion trajectory, and obtaining the pixel coordinates and pixel coordinate changes of the target within the bullet camera's field of view; calculating the PTZ camera rotation parameters based on the pixel coordinate changes and the displacement vector, and generating a PTZ camera pointing vector; extracting the real-time rotation angle from the PTZ camera pointing vector and performing distortion correction processing to obtain the corrected PTZ camera angle; performing consistency verification and fusion processing on the PTZ camera angle and the pixel coordinates to obtain a preliminary transformation matrix; and using the... The initial transformation matrix processes subsequent image frames, extracts the brightness histogram of the current frame, and performs differential analysis to obtain illumination change feature quantities. If the illumination change feature quantities meet preset conditions, they are converted into illumination compensation factors. Based on the illumination compensation factors and real-time acquired device displacement data, the initial transformation matrix is ​​corrected to obtain an optimized transformation matrix. The target's three-dimensional coordinates are mapped to a local coordinate system through the optimized transformation matrix, and the theoretical attitude angle of the target is calculated. The difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera is calculated to obtain the tracking error. If the tracking error is lower than a preset error locking threshold, the automatic tracking mode is locked, and the calibration result of high-precision tracking of the PTZ camera is obtained. This invention achieves high-precision automatic calibration of the PTZ camera by accurately acquiring the optical center displacement vectors of the PTZ camera and combining feature point tracking to calculate the parallax compensation angle to adjust the PTZ camera pointing. It integrates distortion, illumination, and micro-displacement calibration to optimize the transformation matrix, and uses closed-loop verification to lock the tracking mode, thereby improving tracking accuracy and stability.

[0070] Reference Figure 2 The second embodiment of the present invention provides an automatic calibration system for a camera with a PTZ (Panorama) linkage, comprising: The data acquisition module is used to acquire the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun. The feature tracking module is used to extract feature points of the target from the image sequence, perform pyramid optical flow tracking analysis on the feature points, construct discrete motion trajectories, and obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view. The parallax compensation module is used to calculate the PTZ camera rotation parameters and generate the PTZ camera pointing vector based on the pixel coordinate changes and the displacement vector; The matrix generation module is used to extract the real-time rotation angle from the PTZ camera pointing vector and perform distortion removal processing to obtain the corrected PTZ camera angle. The consistency verification and fusion processing of the PTZ camera angle and the pixel coordinates are performed to obtain a preliminary transformation matrix. The illumination compensation module is used to process subsequent image frames using the preliminary transformation matrix, extract the brightness histogram of the current frame and perform differential analysis to obtain illumination change feature quantities. If the illumination change feature quantities meet the preset conditions, the illumination change feature quantities are converted into illumination compensation factors. The matrix optimization module is used to correct the preliminary transformation matrix based on the illumination compensation factor and the real-time acquired equipment displacement data to obtain an optimized transformation matrix; The error calculation module is used to map the three-dimensional coordinates of the target to the local coordinate system through the optimized transformation matrix, calculate the theoretical attitude angle of the target, calculate the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera, and obtain the tracking error. The tracking lock module is used to lock the automatic tracking mode if the tracking error is lower than a preset error lock threshold, so as to obtain the calibration result of the high-precision tracking of the gun-ball linkage.

[0071] It should be noted that the automatic calibration system for a PTZ camera provided in this embodiment of the invention is used to execute all the process steps of the automatic calibration method for a PTZ camera in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0072] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0073] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. An automatic calibration method for a PTZ camera, characterized in that, include: Acquire the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun; Feature points of the target are extracted from the image sequence, and pyramid optical flow tracking analysis is performed on the feature points to construct discrete motion trajectories, thereby obtaining the pixel coordinates and pixel coordinate changes of the target within the gun's field of view. Based on the pixel coordinate changes and the displacement vector, the PTZ camera rotation parameters are calculated and the PTZ camera pointing vector is generated; The real-time rotation angle is extracted from the PTZ camera pointing vector and distortion removal is performed to obtain the corrected PTZ camera angle. The consistency verification and fusion processing of the PTZ camera angle and the pixel coordinates are performed to obtain the preliminary transformation matrix. The initial transformation matrix is ​​used to process subsequent image frames, extract the brightness histogram of the current frame and perform differential analysis to obtain illumination change feature quantities. If the illumination change feature quantities meet the preset conditions, the illumination change feature quantities are converted into illumination compensation factors. The preliminary transformation matrix is ​​corrected based on the illumination compensation factor and the real-time acquired equipment displacement data to obtain an optimized transformation matrix. The target's three-dimensional coordinates are mapped to the local coordinate system through the optimized transformation matrix, the theoretical attitude angle of the target is calculated, and the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera is calculated to obtain the tracking error. If the tracking error is lower than the preset error locking threshold, the automatic tracking mode is locked to obtain the calibration result of the high-precision tracking of the gun-ball linkage.

2. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, The acquisition of the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun includes: The camera captures a continuous sequence of images of the monitored area, and the camera obtains the three-dimensional coordinates of the target within the monitored area and the point cloud data of the camera and PTZ camera shells through a three-dimensional coordinate measuring device. Based on the point cloud data of the outer shell, point cloud registration is performed in combination with the preset equipment design parameters to calculate the spatial coordinates of the optical center of the bullet camera and the optical center of the PTZ camera, and the vector difference between the spatial coordinates of the optical center of the bullet camera and the optical center of the PTZ camera is calculated to generate the initial displacement vector. The attitude quaternions of the camera and the PTZ camera are obtained. Based on a preset transformation formula, the attitude quaternions are converted into a relative rotation matrix. The initial displacement vector is rotated using the relative rotation matrix to obtain the displacement vector between the optical centers of the camera and the PTZ camera.

3. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, The process of extracting target feature points from the image sequence, performing pyramid optical flow tracking analysis on the feature points, constructing discrete motion trajectories, and obtaining the target's pixel coordinates and pixel coordinate changes within the gun's field of view includes: Pixel grayscale gradient analysis is performed on the first frame of the image sequence to identify pixels with corner features as initial target feature points; The initial target feature points are input into a preset pyramid optical flow tracking model to calculate the predicted position coordinates of the initial target feature points in the current frame image coordinate system. Based on the reverse optical flow tracking error of the predicted position coordinates, feature points with errors within a preset range are selected to obtain accurate feature points; The precise feature points are sorted according to the timestamps to construct a discrete motion trajectory point series. Polynomial curve fitting is performed on the motion trajectory point series to obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view.

4. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, The step of calculating the PTZ camera rotation parameters and generating the PTZ camera pointing vector based on the pixel coordinate changes and the displacement vector includes: By combining the focal length of the PTZ camera with the position of the optical center, the pixel coordinate changes and the displacement vector are mapped into a spatial motion vector; Obtain the baseline distance between the bullet camera and the PTZ camera, and calculate the parallax compensation angle based on the geometric relationship between the spatial motion vector and the baseline distance; Compare the parallax compensation angle with the preset parallax judgment threshold. If it does not exceed the threshold, directly call the current preset PTZ camera rotation parameters as PTZ camera rotation data, and generate the PTZ camera pointing vector based on the PTZ camera rotation data. If the error exceeds the limit, the horizontal and vertical parallax compensation angles are calculated based on the horizontal and vertical components of the pixel coordinate changes, and the horizontal and vertical rotation angles of the PTZ camera are corrected to obtain the parallax-corrected PTZ camera rotation data. An adjusted PTZ camera pointing vector is then generated based on the PTZ camera rotation data.

5. The automatic calibration method for a PTZ camera according to claim 4, characterized in that, The process involves extracting the real-time rotation angle from the PTZ camera's pointing vector and performing distortion correction to obtain the corrected PTZ camera angle. Then, a consistency check and fusion process is performed between the PTZ camera angle and the pixel coordinates to obtain a preliminary transformation matrix, including: The PTZ camera pointing vector is analyzed to extract the real-time rotation angle, which includes the horizontal azimuth angle and the vertical pitch angle; The real-time rotation angle is corrected by calling a preset PTZ camera lens distortion model to obtain the corrected PTZ camera angle. Based on the preset internal parameters of the camera, the displacement vector, and the baseline distance, the theoretical projection pixel coordinates of the PTZ camera angle on the camera image plane are calculated. Calculate the reprojection error value between the theoretical projected pixel coordinates and the pixel coordinates. If the reprojection error value meets the preset consistency constraint condition, it is determined that the two correspond. The corrected PTZ camera angle and the pixel coordinates are fused to generate a preliminary transformation matrix containing rotation and translation components. If the reprojection error value does not meet the preset consistency constraint, it is determined that the two do not correspond, the current feature point is discarded, and the process returns to the step of extracting the feature points of the target.

6. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, The process involves using the preliminary transformation matrix to process subsequent image frames, extracting the current frame's brightness histogram, and performing differential analysis to obtain illumination change features. If the illumination change features meet preset conditions, they are then converted into illumination compensation factors, including: The initial transformation matrix is ​​used to perform spatial mapping processing on subsequent image frames to separate the brightness component of the image, and a grayscale distribution map is constructed based on the brightness component. Statistically analyze the percentage of pixels with different brightness values ​​in the grayscale distribution map to generate a brightness histogram for the current frame. Calculate the difference between the current frame brightness histogram and the preset reference frame brightness histogram, and accumulate the difference by weighting to obtain the global illumination change feature. The illumination change characteristic quantity is compared with the preset illumination change threshold. If the preset condition is met, the illumination change characteristic quantity is converted into an illumination compensation factor according to a preset ratio. If the condition is not met, the illumination compensation factor is set to a preset benchmark value.

7. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, The process of refining the initial transformation matrix based on the illumination compensation factor and real-time acquired equipment displacement data to obtain an optimized transformation matrix includes: The illumination compensation factor is weighted and calculated with each component of the preliminary transformation matrix to generate an intermediate transformation matrix; Alignment analysis is performed between the real-time mechanical displacement data output by the displacement monitoring sensor and the spatial reference of the intermediate state transformation matrix to calculate the device displacement vector containing direction and amplitude. Compare the amplitude of the device displacement vector with the preset micro-motion detection threshold. If it does not exceed the threshold, it is determined that no displacement calibration is required, and the intermediate state transformation matrix is ​​directly used as the optimized transformation matrix. If the value exceeds the limit, the spatial position offset is calculated based on the device displacement vector. An adaptive parameter set containing horizontal offset correction values ​​and vertical offset correction values ​​is constructed. The adaptive parameter set is then superimposed on the intermediate state transformation matrix for a second iteration update to obtain the optimized transformation matrix.

8. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, The process of mapping the target's three-dimensional coordinates to a local coordinate system using the optimized transformation matrix, calculating the target's theoretical attitude angle, and calculating the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera to obtain the tracking error includes: The optimized transformation matrix is ​​used to map the target's three-dimensional coordinates to the PTZ camera's local coordinate system; Based on the target position in the local coordinate system of the PTZ camera, the theoretical azimuth angle and theoretical pitch angle are calculated, which together form the theoretical attitude angle of the target; The PTZ camera receives actual attitude angle data fed back by its built-in sensors, and the actual attitude angle data includes the actual azimuth angle and the actual pitch angle. The difference between the actual azimuth angle and the theoretical azimuth angle, and the difference between the actual pitch angle and the theoretical pitch angle are calculated respectively. The two sets of differences are integrated into a tracking error signal, and the tracking error corresponding to the tracking error signal is extracted.

9. The automatic calibration method for a PTZ camera according to claim 1, characterized in that, If the tracking error is lower than a preset error locking threshold, then the automatic tracking mode is locked to obtain the calibration result of the high-precision tracking of the ball-and-gun linkage, including: Compare the tracking error with a preset error lockout threshold. If the error is not lower than the error lockout threshold, return to re-execute the step of calculating the disparity compensation angle based on the pixel coordinate change and the displacement vector, and the subsequent steps. If the error is below the aforementioned error lockout threshold, a lockout signal for automatic tracking mode is generated, and the servo control parameters of the PTZ camera are fixed based on the lockout signal. Based on the servo control parameters, the PTZ camera is controlled to follow the target movement, and the calibration result of high-precision tracking with gun-ball linkage is obtained.

10. An automatic calibration system for a PTZ camera, characterized in that, include: The data acquisition module is used to acquire the target's three-dimensional coordinates, the displacement vector between the optical centers of the gun and the PTZ camera, and the image sequence of the gun. The feature tracking module is used to extract feature points of the target from the image sequence, perform pyramid optical flow tracking analysis on the feature points, construct discrete motion trajectories, and obtain the pixel coordinates and pixel coordinate changes of the target within the gun's field of view. The parallax compensation module is used to calculate the PTZ camera rotation parameters and generate the PTZ camera pointing vector based on the pixel coordinate changes and the displacement vector; The matrix generation module is used to extract the real-time rotation angle from the PTZ camera pointing vector and perform distortion removal processing to obtain the corrected PTZ camera angle. The consistency verification and fusion processing of the PTZ camera angle and the pixel coordinates are performed to obtain a preliminary transformation matrix. The illumination compensation module is used to process subsequent image frames using the preliminary transformation matrix, extract the brightness histogram of the current frame and perform differential analysis to obtain illumination change feature quantities. If the illumination change feature quantities meet the preset conditions, the illumination change feature quantities are converted into illumination compensation factors. The matrix optimization module is used to correct the preliminary transformation matrix based on the illumination compensation factor and the real-time acquired equipment displacement data to obtain an optimized transformation matrix; The error calculation module is used to map the three-dimensional coordinates of the target to the local coordinate system through the optimized transformation matrix, calculate the theoretical attitude angle of the target, calculate the difference between the theoretical attitude angle and the actual attitude angle fed back by the PTZ camera, and obtain the tracking error. The tracking lock module is used to lock the automatic tracking mode if the tracking error is lower than a preset error lock threshold, so as to obtain the calibration result of the high-precision tracking of the gun-ball linkage.

Images