A hierarchical decomposition-based gun-ball linkage calibration method and system, and a storage medium

Abstract

The application discloses a kind of gun ball linkage calibration method, system and storage medium based on hierarchical decomposition, method includes: obtaining single video frame, according to the single video frame, pre-construct and calibrate splicing model, obtain spliced panoramic video frame;Obtain the gimbal video frame, according to the gimbal video frame and single video frame, pre-construct and calibrate homography model and pinhole model, to complete gun ball linkage calibration;According to the coordinate of the panoramic video frame, coordinate conversion is carried out based on splicing model, and single video pixel coordinate is obtained;According to the single video pixel coordinate, coordinate conversion is carried out based on homography model, and gimbal video PTZ rough orientation coordinate is obtained;According to the gimbal video PTZ rough orientation coordinate, coordinate conversion is carried out based on pinhole model, and gimbal video PTZ accurate orientation coordinate is obtained, to complete gun ball linkage calculation.The application realizes the ball linkage calibration and accurate gun ball linkage control between remote tower system panoramic video and gimbal video.

Description

Technical Field

[0001] This invention relates to the field of gun-ball linkage calibration technology, specifically to a gun-ball linkage calibration method, system, and storage medium based on hierarchical decomposition. Background Technology

[0002] As the core of the remote control tower system, the optical system provides remote central controllers with real-time monitoring images of the remote airport in the form of video. To meet the remote central controllers' requirements for airport safety, the remote control tower system generally provides two types of video: a panoramic view of the airport, used to display overall situational information; and a region of interest (GPI) video, used to display detailed information about key areas. The system uses camera-to-body (PTZ) linkage technology to achieve joint calibration and interactive communication between the panoramic and PTP videos, with camera-to-body joint calibration being the foundation for this interactive communication.

[0003] The joint camera-to-body (PTZ) calibration technique is used to establish the mapping relationship between the panoramic video frame coordinate system and the PTZ camera coordinate system. Traditional camera-to-body linkage methods are based on specific camera models, such as pinhole camera models that conform to optical distortion or wide-angle camera models that conform to mirror distortion. Image compensation is performed using optical principles before camera-to-body linkage calculations are conducted. However, because other types of distortion are introduced during the panoramic image stitching process, which does not conform to existing camera models, directly applying the above calibration methods to stitched panoramic images will lead to reduced coordinate transformation accuracy, resulting in visible errors or even erroneous results.

[0004] The camera-ball linkage method for stitched panoramas includes two main steps: calibration data acquisition and calibration model optimization. Calibration data acquisition first obtains the coordinate points on the panoramic image. Then place the center point of the pan-tilt-zoom (PTZ) video feed physically opposite to the center point of the camera. Acquire the gimbal camera at the same location. The coordinates are used to form a set of matching point pairs. Multiple sets of matching point pairs are obtained as input to the calibration model optimization module. The optimization methods used are generally nearest neighbor, K-nearest neighbor, etc. However, the above methods require a large number of calibration point pairs, which leads to the problem of time-consuming and labor-intensive camera calibration. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a gun-ball linkage calibration method, system, and storage medium based on hierarchical decomposition.

[0006] Firstly, a gun-ball linkage calibration method based on hierarchical decomposition includes:

[0007] Acquire a single video frame, pre-construct and calibrate a stitching model based on the single video frame, and obtain stitched panoramic video frames;

[0008] Acquire PTZ video frames, and pre-construct and calibrate homography and pinhole models based on the PTZ video frames and single-channel video frames to complete the PTZ linkage calibration;

[0009] Based on the coordinates of the panoramic video frames, coordinate transformation is performed using a stitching model to obtain the pixel coordinates of a single video stream;

[0010] Based on the single-channel video pixel coordinates, coordinate transformation is performed using the homography model to obtain the approximate orientation coordinates of the PTZ video.

[0011] Based on the rough orientation coordinates of the PTZ video, coordinate transformation is performed using a pinhole model to obtain the precise orientation coordinates of the PTZ video, thus completing the gun-ball linkage calculation.

[0012] Furthermore, the step of acquiring a single video frame, and pre-constructing and calibrating a stitching model based on the single video frame to obtain stitched panoramic video frames specifically involves:

[0013] A splicing model is constructed, which includes a perspective transformation model and an image correction model. The image correction model includes a spherical projection model, an Euler angle rotation model, and a back projection model.

[0014] Acquire single-channel video frames captured by the camera, and obtain the perspective transformation matrix between any adjacent single-channel video frames based on the perspective transformation model to calibrate the perspective transformation model.

[0015] The initial panoramic image coordinates are calculated based on the calibrated perspective transformation model, and the stitched initial panoramic video frames are obtained.

[0016] Obtain the center point of the initial panoramic video frame, transform the image coordinates corresponding to the initial panoramic video frame to a three-dimensional plane and obtain the three-dimensional plane coordinates, calibrate the spherical projection model according to the three-dimensional plane coordinates, and obtain the spherical coordinates of the initial panoramic video frame based on the spherical projection model according to the three-dimensional plane coordinates.

[0017] The rotation matrix is ​​obtained based on the spherical coordinates of the initial panoramic video frame to calibrate the Euler angle rotation model, and the rotation spherical coordinates are obtained based on the Euler angle rotation model.

[0018] The back projection model is calibrated based on the center point of the initial panoramic video frame, and the corrected panoramic video frame coordinates are obtained based on the back projection model according to the rotating spherical coordinates, so as to obtain the final stitched panoramic video frame.

[0019] Furthermore, the acquisition of PTZ video frames, and the pre-construction and calibration of homography and pinhole models based on the PTZ video frames and single-channel video frames to complete the PTZ linkage calibration, include:

[0020] Based on the single-channel video frame collection For each clearly defined corner point in an image, obtain the corresponding single-channel video corner point coordinates;

[0021] Acquire the PTZ video frames captured by the PTZ camera, control the PTZ camera according to the coordinates of each single-channel video corner point, so that the center point of the PTZ video coincides with the corresponding corner point, and obtain the PTZ camera orientation coordinate value when they coincide.

[0022] By combining the coordinates of the corner points in the single-channel video feed with the orientation coordinates of the corresponding PTZ camera, a corresponding point is established. To construct a homography model, the homography mapping relationship between point pairs is used.

[0023] The homography matrix was estimated using the least squares method to calibrate the homography model.

[0024] Furthermore, the step of acquiring PTZ video frames, and pre-constructing and calibrating homography and pinhole models based on the PTZ video frames and single-channel video frames to complete the PTZ linkage calibration, also includes:

[0025] Obtain the parameters of the PTZ camera, and obtain the PTZ video frame parameters based on the PTZ video frames;

[0026] A pinhole model is constructed based on the parameters of the PTZ camera and the parameters of the PTZ video frame. The model parameters of the pinhole model include: the coordinates of the center point of the PTZ video, the horizontal and vertical field of view of the PTZ camera, and the width and height of the PTZ video frame.

[0027] Furthermore, the step of performing coordinate transformation based on the stitching model using the coordinates of the panoramic video frame to obtain the pixel coordinates of a single video stream specifically involves:

[0028] Based on the back projection model, the corrected panoramic video frame coordinates are transformed to a three-dimensional plane to obtain back-projected three-dimensional coordinates. Then, the panoramic video frame is projected onto a spherical coordinate system according to the back-projected three-dimensional coordinates to obtain the rotated spherical coordinates of the panoramic video frame.

[0029] Based on the Euler angle rotation model, the spherical coordinates of the initial panoramic video frame are obtained by applying the rotation spherical coordinates to the inverse of the rotation matrix.

[0030] Based on the spherical projection model, the spherical coordinates are transformed into a three-dimensional plane to obtain the spherical projection three-dimensional coordinates, and the spherical projection three-dimensional coordinates are transformed into a two-dimensional plane to obtain the initial panoramic image coordinates;

[0031] Based on the perspective transformation model, the initial panoramic image coordinates are converted into single-channel video pixel coordinates according to the perspective transformation matrix. The conversion formula is as follows:

[0032] ,

[0033] Among them, the The ref represents the perspective transformation matrix between any two adjacent single video streams, and the ref represents the panoramic image reference video frame. Indicates the first video frames, Represents the initial panoramic image coordinates. Indicates the first Video pixel coordinates.

[0034] Furthermore, the step of performing coordinate transformation based on the homography model using the single-channel video pixel coordinates to obtain the approximate PTZ orientation coordinates of the PTZ video is as follows:

[0035] Based on the homography model, the pixel coordinates of a single video stream are converted into the approximate orientation coordinates of the PTZ (pan-tilt-zoom) video stream. The conversion formula is as follows:

[0036] ,

[0037] Among them, the The approximate orientation coordinates for the PTZ camera video. It is a homography matrix.

[0038] Furthermore, the step of performing coordinate transformation based on the pinhole model using the coarse orientation coordinates of the PTZ video to obtain the precise orientation coordinates of the PTZ video, in order to complete the gun-ball linkage calculation, specifically involves:

[0039] The relative motion angle of the PTZ camera is obtained based on the pinhole model, and the calculation formula is as follows:

[0040] ;

[0041] Based on the relative motion angle, the approximate orientation coordinates of the PTZ video are transformed to obtain the precise orientation coordinates of the PTZ video, thus completing the gun-ball linkage calculation. The calculation formula is as follows:

[0042] ;

[0043] Among them, the The relative motion angle of the pan-tilt camera, the The approximate orientation coordinates of the PTZ video are given. For precise orientation coordinates of the PTZ camera in pan-tilt-zoom (PTZ) mode, the... The coordinates of the center point of the PTZ video are... The preset coordinates of the precise target center point for the PTZ video are... and These refer to the horizontal and vertical field of view of the pan-tilt camera, respectively. and These are the width and height of the PTZ video frame, respectively.

[0044] Secondly, a gun-ball linkage calibration method system based on hierarchical decomposition includes:

[0045] First linkage calibration module: used to acquire single video frames, pre-build and calibrate the stitching model based on the single video frames, and obtain stitched panoramic video frames;

[0046] The second linkage calibration module is used to acquire PTZ video frames, and to pre-build and calibrate homography and pinhole models based on the PTZ video frames and single-channel video frames in order to complete the PTZ linkage calibration.

[0047] Single-channel video restoration module: used to perform coordinate transformation based on the stitching model according to the coordinates of the panoramic video frame to obtain the pixel coordinates of the single-channel video;

[0048] The PTZ orientation estimation module is used to perform coordinate transformation based on the homography model according to the pixel coordinates of the single video stream to obtain the approximate orientation coordinates of the PTZ video.

[0049] Precise Gimbal Orientation Module: This module is used to perform coordinate transformation based on the pinhole model using the coarse orientation coordinates of the PTZ video to obtain the precise orientation coordinates of the PTZ video, thereby completing the gun-ball linkage calculation.

[0050] Thirdly, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in the first aspect.

[0051] The beneficial effects of this invention are as follows: Based on single-channel video frames, a stitching model, a homography model, and a pinhole model are constructed and calibrated to achieve gun-PTZ linkage calibration. Based on the models, stitched panoramic video frames are obtained. Simultaneously, a hierarchical decomposition strategy is introduced. The stitching model converts panoramic image coordinates into single-channel video pixel coordinates. The homography model converts single-channel video pixel coordinates into coarse orientation coordinates in the PTZ coordinate system of the PTZ camera. The pinhole model converts the coarse orientation coordinates in the PTZ coordinate system into precise orientation coordinates. This decomposes the complex conversion model between panoramic image coordinates and PTZ coordinates into a multi-stage simple model, enabling rapid and precise gun-PTZ linkage control between panoramic video and PTZ video in a remote control tower system. It also enables precise PTZ 3D positioning of targets in panoramic video, assisting controllers in efficiently monitoring airport targets and significantly improving airport surface security capabilities. Attached Figure Description

[0052] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0053] Figure 1 A conceptual framework diagram of a gun-ball linkage calibration method based on hierarchical decomposition provided in an embodiment of the present invention;

[0054] Figure 2 A flowchart of a gun-ball linkage calibration method based on hierarchical decomposition provided in an embodiment of the present invention;

[0055] Figure 3 This is a block diagram of a gun-ball linkage calibration system based on hierarchical decomposition, provided for an embodiment of the present invention. Detailed Implementation

[0056] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0057] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0058] The inventive concept of this embodiment is to introduce a hierarchical decomposition strategy to decompose the complex transformation model between panoramic image coordinates and PTZ coordinates of the gimbal into a multi-stage simple model, thereby reducing the number of calibration point pairs required for the mapping model, improving the calibration efficiency of the gun-ball linkage camera in engineering applications, and improving the accuracy and precision of coordinate mapping.

[0059] like Figure 1 As shown, a gun-ball mapping model is established to realize the pixel points in the panoramic video frame coordinate system. Orientation in the PTZ coordinate system of the gimbal video The conversion between these models, including the splicing model, homography model, and pinhole model, and the inventive concept framework, include:

[0060] (1) Pixel coordinates in the panoramic video pixel coordinate system →Pixel coordinates in a single-channel video pixel coordinate system The two can be converted through a splicing model;

[0061] (2) Pixel coordinates in a single-channel video pixel coordinate system → Rough orientation coordinates in the PTZ coordinate system of the gimbal video The two can be converted using a homography model;

[0062] (3) Coarse orientation coordinates in the PTZ coordinate system of the PTZ video camera →Precise orientation coordinates in the PTZ coordinate system of the PTZ video camera The two are converted using a pinhole model.

[0063] Based on the above inventive concept, this embodiment proposes a gun-ball linkage calibration method based on hierarchical decomposition, such as... Figure 2 As shown, it includes:

[0064] S1: Obtain a single video frame, pre-build and calibrate a stitching model based on the single video frame, and obtain stitched panoramic video frames.

[0065] Specifically, a stitching model is constructed, which includes an image correction model and a perspective transformation model. The image correction model includes a spherical projection model, an Euler angle rotation model, and a back projection model. Based on the constructed stitching model, preprocessing is performed to obtain the stitched panoramic video frames and the parameter values ​​of the stitching model. The preprocessing process includes:

[0066] S101: Acquire a single video frame captured by the camera, and obtain the perspective transformation matrix between any adjacent single video frames based on the perspective transformation model to calibrate the perspective transformation model.

[0067] Specifically, the system acquires single-channel video frames captured by the camera. These N single-channel video frames are then grouped into pairs based on their physical adjacency. The two video frames in each group share a partial overlap. Two video frames from the same time point are extracted, and feature extraction and feature matching are performed on the extracted video frames to obtain the matching points for each group of video frames. Preferably, this embodiment uses the deep learning-based feature descriptor LIFT (Learned Invariant Feature Transform) for feature extraction and the UCN (Universal Correspondence Network) method for feature matching.

[0068] It should be noted that, in addition to the LIFT method, classic feature descriptors such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and FAST (Oriented FAST and Rotated BRIEF) can also be considered for feature extraction. In addition to the UCN method, RANSAC (Random Sample Consensus) and MatchNet can also be used for feature matching, and no restrictions are imposed here.

[0069] Assuming adjacent video frames and video frames The corresponding matching points are as follows:

[0070]

[0071] in, Indicates the number of matching points. and They represent the first Road video pixel coordinates and the The pixel coordinates of the video stream, for the k-th matching point pair, can be represented as:

[0072] (1)

[0073] in, This represents the proportionality coefficient. The perspective transformation matrix between any two adjacent single video streams is a 3×3 homogeneous matrix, estimated using the least squares method.

[0074] The perspective transformation model is calibrated using the perspective transformation matrix. Assume... For reference frame, the first The video frames are transformed by the perspective transformation matrix. Switch to In the coordinate system, based on the transitive properties of the perspective transformation matrix, we can obtain... Based on the perspective transformation matrix, the reference frame and the Road video frames The relationship between them can be described by formula (2), that is, the calibrated perspective transformation model includes:

[0075] (2)

[0076] in, Let be the perspective transformation matrix between any two adjacent single video streams. Use the panoramic image as a reference video frame. For the first video frames, For the first The video pixel coordinates are used in this embodiment, with the first video frame as the reference frame. ,therefore Equivalent to coordinates in the initial panoramic image coordinate system .

[0077] Based on the single-channel video frames captured by the camera, the pixel coordinates of the single-channel video are... Substituting into formula (2), the coordinates of the initial panoramic image can be calculated. This process yields the initial panoramic video frames for stitching.

[0078] S102: Obtain the center point of the initial panoramic video frame, transform the image coordinates corresponding to the initial panoramic video frame to a three-dimensional plane and obtain the three-dimensional plane coordinates, calibrate the spherical projection model according to the three-dimensional plane coordinates, and obtain the spherical coordinates of the initial panoramic video frame based on the spherical projection model according to the three-dimensional plane coordinates.

[0079] Specifically, obtain the center point of the initial panoramic video frame to be stitched together. Establish with A spatial coordinate system with the origin (i.e., the center point of the sphere) subscript express The initial panoramic image coordinates are rotated to a new coordinate system based on a spherical projection model. In On a plane, It is the radius of the sphere, and also the center point of the initial panoramic video frame. Translate to coordinate system midpoint Therefore, the coordinates of any initial panoramic image on the stitched initial panoramic video frame... to plane The conversion relationship is as follows:

[0080] (3)

[0081] In the formula, the subscript express superscript This represents the initial panoramic video frame. These are the three-dimensional planar coordinates of the panoramic video frame.

[0082] Based on the spherical projection model, the coordinate system mid plane Three-dimensional plane coordinates of any point on Transform to a sphere to obtain the spherical coordinates of the initial panoramic video frame. The transformation relationship is as follows:

[0083] (4)

[0084] Among them, subscript express .

[0085] The spherical projection model includes formulas (3) and (4). In this embodiment, the center point of the panoramic video frame is determined. and The values ​​are substituted into formulas (3) and (4) to complete the calibration of the spherical projection model, and then the initial panoramic image coordinates are used to determine the model. The spherical coordinates of the initial panoramic video frame are obtained based on the spherical projection model. .

[0086] S103: Obtain the rotation matrix based on the spherical coordinates of the initial panoramic video frame to calibrate the Euler angle rotation model, and obtain the rotation spherical coordinates based on the Euler angle rotation model;

[0087] Specifically, assuming the spatial coordinate system in step S102 of The rotation directions of the axes are defined as follows: Based on the Euler angle rotation model, any point on the sphere Around Rotate in different directions Then, the corresponding rotation matrix is ​​obtained. :

[0088] (5)

[0089] any point on the sphere By rotation matrix After transformation, we obtain the spherical coordinates of revolution. The calibrated Euler angle rotation model is as follows:

[0090] (6)

[0091] In the formula, Middle and upper bids express .

[0092] Based on the calibrated Euler angle rotation model, the spherical coordinates of the initial panoramic video frame are... Substituting into formula (6) yields the coordinates of the sphere of revolution. .

[0093] S104: The back projection model is calibrated based on the center point of the initial panoramic video frame, and the corrected panoramic video frame coordinates are obtained based on the back projection model according to the rotating spherical coordinates, so as to obtain the final stitched panoramic video frame.

[0094] Specifically, based on the back projection model, the rotating spherical coordinates Back projection onto coordinate system middle On the plane, the corresponding corrected three-dimensional plane coordinates are obtained. Then projected onto the panoramic image coordinate system This is the inverse operation of step S102. Let... Based on the back projection model, according to the formula:

[0095] (7)

[0096] We can obtain:

[0097] (8)

[0098] The three-dimensional plane coordinates obtained above The corrected panoramic video frame coordinates are obtained by conversion using formula (9). :

[0099] (9)

[0100] In the formula, These are the coordinates of the center point of the panoramic video frame.

[0101] The back projection model includes formulas (7)-(9). In this embodiment, the center point of the initial panoramic video frame is determined. and the radius of the sphere The values ​​can be substituted into formulas (7)-(9) to complete the calibration of the back projection model, and then the values ​​can be determined based on the coordinates of the rotating spherical surface. The corrected panoramic video frame coordinates are obtained based on the back projection model. And obtain the final stitched panoramic video frames.

[0102] S2: Acquire PTZ video frames, and pre-construct and calibrate homography and pinhole models based on the PTZ video frames and single-channel video frames to complete the PTZ linkage calibration.

[0103] Specifically, the homography model includes a homography matrix, which describes the pixel coordinates in a single-channel video pixel coordinate system. Rough orientation coordinates in the PTZ coordinate system of the gimbal video The conversion relationship between them.

[0104] Collecting single-channel video in ground areas For each corner point in the image with clearly defined features, obtain the corresponding single-channel video corner point coordinates. Acquire pan-tilt-zoom (PTZ) video frames captured by the PTZ camera. For each corner point, control the PTZ camera so that the center point of the PTZ video coincides with the corresponding corner point, and obtain the corresponding PTZ camera orientation coordinates at this moment. The value of .

[0105] The coordinates of the corner point in the single-channel video and the orientation coordinates of the corresponding PTZ camera at the corner point are used. The values ​​form a set of corresponding points, establishing homography mapping relationship between point pairs To construct a homography model, the least squares method is used to estimate the homography matrix to calibrate the homography model. The homography matrix is ​​a 3×3 matrix, which can be represented as:

[0106] (10)

[0107] Furthermore, the homography model, based on the homography assumption, has the advantage of requiring only four or more reference point pairs for calibration. However, when the center of the panoramic video is not on the line connecting the optical center of the gimbal lens and the imaging plane, the greater the distance from the center of the panoramic video, the greater the linkage error, meaning the distance between the gimbal video center point and the pixel coordinates is relatively large. To solve the above problems, this embodiment obtains a more accurate target center position through manual interaction and introduces a local pinhole model to achieve precise gimbal control. The pinhole model describes the coarse orientation coordinates in the gimbal video PTZ coordinate system. Precise orientation coordinates in the PTZ coordinate system of the gimbal video The conversion relationship between them.

[0108] Establish the current gimbal camera coordinate system, where the optical center of the gimbal camera is the origin, corresponding to... The axes are parallel to the imaging plane and are respectively parallel to the coordinates in the image coordinate system. Consistent direction The axis is perpendicular to the program plane. (Assume...) , Therefore, this embodiment transforms the problem of determining the accurate orientation of the gimbal camera into solving the relative motion angle of the gimbal. .

[0109] Specifically, the parameter information of the PTZ camera and the parameter information of the PTZ video frame are obtained. The PTZ camera parameter information includes, but is not limited to, the horizontal and vertical field of view of the PTZ camera. The PTZ video frame parameter information includes, but is not limited to, the coordinates of the PTZ video center point, the preset coordinates of the precise target center point of the PTZ video, and the width and height of the PTZ video frame.

[0110] Assuming the pan-tilt camera has a focal length multiplier of 1, a pinhole model is constructed using the principle of similar triangles based on the pan-tilt camera parameters and the pan-tilt video frame parameters. The pinhole model parameters include: the coordinates of the center point of the pan-tilt video, the horizontal and vertical field of view of the pan-tilt camera, and the width and height of the pan-tilt video frame.

[0111] Steps S1 and S2 complete the construction of the splicing model, homography model, and pinhole model, and realize the gun-ball linkage calibration.

[0112] S3: Based on the panoramic video frames, perform coordinate transformation using the stitching model to obtain the pixel coordinates of a single video stream;

[0113] Specifically, such as Figure 1As shown, based on the gun-ball linkage calibration results in steps S1 and S2, the pixel coordinates in the panoramic video frame coordinate system are known. Inverse calculation of pixel coordinates in a single-channel video pixel coordinate system Specifically, it includes:

[0114] S301: Based on the back projection model and the corrected panoramic video frame coordinates Using formula (9), the panoramic video frame is placed on the three-dimensional plane to obtain the back-projected three-dimensional coordinates. Based on the back-projection three-dimensional coordinates, the panoramic video frame is projected onto the spherical coordinate system using formula (7) based on the back-projection model, thus obtaining the rotated spherical coordinates of the panoramic video frame. .in, .

[0115] S302: Based on the Euler angle rotation model, formula (6) is used to rotate the spherical coordinates. Acting on rotation matrix The inverse matrix is ​​used to obtain the spherical coordinates of the initial panoramic video frame. .

[0116] S303: Based on the spherical projection model, use formula (4) to spherical coordinates Transform to a 3D plane to obtain the 3D coordinates of the spherical projection. And use formula (3) to project the three-dimensional coordinates of the spherical surface. Transform to a two-dimensional plane to obtain the initial panoramic image coordinates. .

[0117] S304: Assuming a reference frame and video frames Through perspective transformation matrix The converted frame is used as a reference frame. Consistent with the panoramic video coordinate system, the pixel coordinates in the panoramic video frame are applied to... The pixel coordinates in the single-channel video pixel coordinate system can be obtained, that is, based on the perspective transformation model, formula (2) is used to transform the uncorrected initial panoramic image coordinates according to the perspective transformation matrix. Convert to single-channel video pixel coordinates :

[0118] (11)

[0119] S4: Based on the single-channel video pixels, perform coordinate transformation using the homography model to obtain the approximate orientation coordinates of the PTZ video of the pan-tilt unit.

[0120] Specifically, according to formula (10), the pixel coordinates of a single video stream are determined based on the homography matrix. Convert to the PTZ coordinate system of the PTZ video for a rough orientation coordinate The conversion formula is:

[0121] (12)

[0122] In the formula, The approximate orientation coordinates for the PTZ camera video. It is a homography matrix.

[0123] S5: Based on the rough orientation coordinates of the PTZ video, coordinate transformation is performed using the pinhole model to obtain the precise orientation coordinates of the PTZ video, so as to complete the gun-ball linkage.

[0124] Specifically, the relative motion angle of the pan-tilt camera is obtained based on the pinhole model. And according to the relative motion angle Roughly orient the PTZ camera of the gimbal to the coordinates. Perform coordinate transformation to obtain the precise orientation coordinates of the PTZ video. Ultimately, this achieves pixel-level mapping in the panoramic video frame coordinate system. Orientation in the PTZ coordinate system of the gimbal video The conversion between them completes the gun-ball linkage, and the calculation formula is as follows:

[0125]

[0126] (13)

[0127] In the formula, The relative motion angle of the pan-tilt camera. The approximate orientation coordinates for the PTZ camera video. To provide precise orientation coordinates for PTZ video, The coordinates of the center point of the PTZ video feed are: The subscript is used to define the precise center point coordinates of the pan-tilt-zoom (PTZ) video target. They represent , These refer to the horizontal and vertical field of view of the pan-tilt-zoom (PTZ) camera, respectively. and These are the width and height of the PTZ video frame, respectively.

[0128] Based on the same inventive concept, embodiments of the present invention provide a gun-ball linkage calibration method system based on hierarchical decomposition, such as... Figure 3 As shown, it includes:

[0129] First linkage calibration module: used to acquire single video frames, pre-build and calibrate the stitching model based on the single video frames, and obtain stitched panoramic video frames;

[0130] The second linkage calibration module is used to acquire PTZ video frames, and to pre-build and calibrate homography and pinhole models based on the PTZ video frames and single-channel video frames in order to complete the PTZ linkage calibration.

[0131] Single-channel video restoration module: used to perform coordinate transformation based on the stitching model according to the coordinates of the panoramic video frame to obtain the pixel coordinates of the single-channel video;

[0132] The PTZ orientation estimation module is used to perform coordinate transformation based on the homography model according to the pixel coordinates of the single video stream to obtain the approximate orientation coordinates of the PTZ video.

[0133] Precise Gimbal Orientation Module: This module is used to perform coordinate transformation based on the pinhole model using the coarse orientation coordinates of the PTZ video to obtain the precise orientation coordinates of the PTZ video, thereby completing the gun-ball linkage calculation.

[0134] It should be noted that for a more detailed workflow of the gun-ball linkage calibration method system based on hierarchical decomposition, please refer to the aforementioned method implementation section, which will not be repeated here.

[0135] This invention constructs and calibrates a stitching model, a homography model, and a pinhole model based on single-channel video frames to achieve PTZ (pan-tilt-zoom) linkage calibration. It acquires stitched panoramic video frames based on these models and introduces a hierarchical decomposition strategy. The stitching model converts panoramic image coordinates to single-channel video pixel coordinates, the homography model converts these coordinates to approximate orientation coordinates in the PTZ coordinate system, and the pinhole model converts these approximate orientation coordinates to precise orientation coordinates. This decomposes the complex conversion model between panoramic image coordinates and PTZ coordinates into multi-stage simpler models, enabling rapid and precise PTZ linkage control between panoramic and PTZ videos in remote control systems. It also achieves accurate PTZ 3D positioning of targets in panoramic video, assisting controllers in efficiently monitoring airport targets and significantly enhancing airport surface security capabilities.

[0136] Furthermore, this embodiment of the invention also provides a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, implement the above-described gun-ball linkage calibration method based on hierarchical decomposition.

[0137] The computer-readable storage medium can be an internal storage unit of the backend server described in the foregoing embodiments, such as the system's hard drive or memory. The computer-readable storage medium can also be an external storage device of the system, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the system. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the system. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0139] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A gun-ball linkage calibration method based on hierarchical decomposition, characterized in that, include: Acquire a single video frame, pre-construct and calibrate a stitching model based on the single video frame, and obtain stitched panoramic video frames; Acquire PTZ video frames, and pre-construct and calibrate homography and pinhole models based on the PTZ video frames and single-channel video frames to complete the PTZ linkage calibration; Based on the coordinates of the panoramic video frames, coordinate transformation is performed using a stitching model to obtain the pixel coordinates of a single video stream; Based on the single-channel video pixel coordinates, coordinate transformation is performed using the homography model to obtain the approximate orientation coordinates of the PTZ video. Based on the rough orientation coordinates of the PTZ video, coordinate transformation is performed using the pinhole model to obtain the precise orientation coordinates of the PTZ video, so as to complete the gun-ball linkage calculation. Specifically, obtaining the stitched panoramic video frames involves: A splicing model is constructed, which includes a perspective transformation model and an image correction model. The image correction model includes a spherical projection model, an Euler angle rotation model, and a back projection model. Acquire single-channel video frames captured by the camera, and obtain the perspective transformation matrix between any adjacent single-channel video frames based on the perspective transformation model to calibrate the perspective transformation model. The initial panoramic image coordinates are calculated based on the calibrated perspective transformation model, and the stitched initial panoramic video frames are obtained. Obtain the center point of the initial panoramic video frame, transform the image coordinates corresponding to the initial panoramic video frame to a three-dimensional plane and obtain the three-dimensional plane coordinates, calibrate the spherical projection model according to the three-dimensional plane coordinates, and obtain the spherical coordinates of the initial panoramic video frame based on the spherical projection model according to the three-dimensional plane coordinates. The rotation matrix is ​​obtained based on the spherical coordinates of the initial panoramic video frame to calibrate the Euler angle rotation model, and the rotation spherical coordinates are obtained based on the Euler angle rotation model. The back projection model is calibrated based on the center point of the initial panoramic video frame, and the corrected panoramic video frame coordinates are obtained based on the back projection model according to the rotating spherical coordinates, so as to obtain the final stitched panoramic video frame. The completion of the gun-ball linkage calibration includes: Based on the single-channel video frame collection n For each clearly defined corner point in an image, obtain the corresponding single-channel video corner point coordinates; Acquire the PTZ video frames captured by the PTZ camera, control the PTZ camera according to the coordinates of each single-channel video corner point, so that the center point of the PTZ video coincides with the corresponding corner point, and obtain the PTZ camera orientation coordinate value when they coincide. By combining the coordinates of the corner points in the single-channel video feed with the orientation coordinates of the corresponding PTZ camera, a corresponding point is established. To construct a homography model, the homography mapping relationship between point pairs is used. The homography matrix was estimated using the least squares method to calibrate the homography model; Obtain the parameters of the PTZ camera, and obtain the PTZ video frame parameters based on the PTZ video frames; A pinhole model is constructed based on the parameters of the PTZ camera and the parameters of the PTZ video frame. The model parameters of the pinhole model include: the coordinates of the center point of the PTZ video, the horizontal and vertical field of view of the PTZ camera, and the width and height of the PTZ video frame.

2. The gun-ball linkage calibration method based on hierarchical decomposition according to claim 1, characterized in that, The step of performing coordinate transformation based on the coordinates of the panoramic video frames using a stitching model to obtain the pixel coordinates of a single video stream is as follows: Based on the back projection model, the corrected panoramic video frame coordinates are transformed to a three-dimensional plane to obtain back-projected three-dimensional coordinates. Then, the panoramic video frame is projected onto a spherical coordinate system according to the back-projected three-dimensional coordinates to obtain the rotated spherical coordinates of the panoramic video frame. Based on the Euler angle rotation model, the spherical coordinates of the initial panoramic video frame are obtained by applying the rotation spherical coordinates to the inverse of the rotation matrix. Based on the spherical projection model, the spherical coordinates are transformed into a three-dimensional plane to obtain the spherical projection three-dimensional coordinates, and the spherical projection three-dimensional coordinates are transformed into a two-dimensional plane to obtain the initial panoramic image coordinates; Based on the perspective transformation model, the initial panoramic image coordinates are converted into single-channel video pixel coordinates according to the perspective transformation matrix. The conversion formula is as follows: ， Among them, the The ref represents the perspective transformation matrix between any two adjacent single video streams, and the ref represents the panoramic image reference video frame. Indicates the first video frames, Represents the initial panoramic image coordinates. Indicates the first Video pixel coordinates.

3. The gun-ball linkage calibration method based on hierarchical decomposition according to claim 2, characterized in that, The step of performing coordinate transformation based on the single-channel video pixel coordinates using a homography model to obtain the approximate orientation coordinates of the PTZ video is as follows: Based on the homography model, the pixel coordinates of a single video stream are converted into the approximate orientation coordinates of the PTZ (pan-tilt-zoom) video stream. The conversion formula is as follows: ， Among them, the The approximate orientation coordinates for the PTZ camera video. It is a homography matrix.

4. The gun-ball linkage calibration method based on hierarchical decomposition according to claim 3, characterized in that, The process involves performing coordinate transformation based on the pinhole model using the approximate orientation coordinates of the PTZ video to obtain the precise orientation coordinates of the PTZ video, thereby completing the gun-ball linkage calculation. Specifically: The relative motion angle of the PTZ camera is obtained based on the pinhole model, and the calculation formula is as follows: ； Based on the relative motion angle, the approximate orientation coordinates of the PTZ video are transformed to obtain the precise orientation coordinates of the PTZ video, thus completing the gun-ball linkage calculation. The calculation formula is as follows: ； Among them, the The relative motion angle of the pan-tilt camera, the The approximate orientation coordinates of the PTZ video are given. For precise orientation coordinates of the PTZ camera in pan-tilt-zoom (PTZ) mode, the... The coordinates of the center point of the PTZ video are... The preset coordinates of the precise target center point for the PTZ video are... and These refer to the horizontal and vertical field of view of the pan-tilt camera, respectively. and These are the width and height of the PTZ video frame, respectively.

5. A gun-ball linkage calibration system based on hierarchical decomposition, characterized in that, The system is applicable to the method as described in any one of claims 1-4, comprising: First linkage calibration module: used to acquire single video frames, pre-build and calibrate the stitching model based on the single video frames, and obtain stitched panoramic video frames; The second linkage calibration module is used to acquire PTZ video frames, and to pre-build and calibrate homography and pinhole models based on the PTZ video frames and single-channel video frames in order to complete the PTZ linkage calibration. Single-channel video restoration module: used to perform coordinate transformation based on the stitching model according to the coordinates of the panoramic video frame to obtain the pixel coordinates of the single-channel video; The PTZ orientation estimation module is used to perform coordinate transformation based on the homography model according to the pixel coordinates of the single video stream to obtain the approximate orientation coordinates of the PTZ video. Precise Gimbal Orientation Module: This module is used to perform coordinate transformation based on the pinhole model using the coarse orientation coordinates of the PTZ video to obtain the precise orientation coordinates of the PTZ video, thereby completing the gun-ball linkage calculation.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-4.

Images