A target linkage tracking system and method based on unmanned aerial vehicle and high-position ball machine video

Abstract

The application discloses a target linkage tracking system and method based on a UAV and a high-position ball machine video, and relates to the technical field of target tracking. The target linkage tracking system comprises a front-end device, an intermediate processing module and a back-end calculation module, the front-end device is connected with the intermediate processing module, and the intermediate processing module is connected with the back-end calculation module. The application uses a high-position ball machine to capture the video of a target area, carries out target recognition analysis on the collected video information, simultaneously links and dispatches a UAV to the scene to collect information, and makes up for the insufficient angle of view of a fixed-position camera. The high-position ball machine continuously monitors a set area for a long time, makes up for the insufficient endurance time of the UAV, and cannot continuously stay above the target to provide information for the back end. The application automatically plans a route, and only needs personnel to meet the conditions to fly, thereby solving the problem that the UAV needs to rely on a well-trained UAV pilot to shoot a video.

Description

Technical Field

[0001] This invention relates to the field of drone and high-position PTZ camera video technology, specifically to a target linkage tracking system and method based on drone and high-position PTZ camera video. Background Technology

[0002] Compared to low-position video surveillance (below 20 meters), high-position PTZ camera video surveillance offers the advantage of a wider field of view, with an exponentially larger coverage area. High-position PTZ cameras equipped with high-speed, high-definition cameras can continuously monitor the surrounding area 24 / 7 from a fixed location. Furthermore, the emergence of drones, with their onboard cameras, compensates for the shortcomings of high-position video surveillance in terms of target angle acquisition. Drones can fly over targets, providing comprehensive video surveillance from all angles without blind spots, acquiring the necessary detailed information. However, drones have a limited battery life, preventing them from continuously loitering over targets to provide information to the backend. The combined use of drones and high-position video surveillance technologies enables large-scale, effective video monitoring that integrates static and dynamic elements and leverages their complementary strengths.

[0003] China Tower Corporation Limited (hereinafter referred to as "Tower Company") possesses the largest number of communication tower resources in China. Utilizing these high-altitude towers to install high-speed, high-definition PTZ cameras is already widespread across various industries. These cameras are extensively deployed throughout the country in areas such as forest and grassland fire prevention, protection of major river basins, road traffic safety, and farmland protection, playing a positive role. With the further development of integrated air-space-ground sensing systems and the rapid development of the low-altitude economy, the "sky eye" linkage method, which combines high-altitude monitoring and drone monitoring in the video surveillance field, will become one of the key requirements for the development of high-quality video surveillance information systems in various industries, based on actual needs.

[0004] High-position video surveillance typically positions cameras with lenses at least 20 meters above the ground, while high-speed PTZ cameras mounted on communication towers are generally around 40 meters high. Drone video surveillance typically operates at altitudes between 80 and 150 meters. Engineering experience shows that the maximum effective target recognition field of view (PDVPS) is generally within 10 times the lens height. For example, a 40-meter-high PTZ camera has a maximum ground PAVPS exceeding 2000 meters, while the effective video pixel range is around 400 meters. In contrast, a drone's PAVPS at a flight altitude of 120 meters has a maximum PAVPS exceeding 6000 meters, with an effective video pixel range of around 1200 meters. Therefore, beyond a certain distance and when the viewing angle is unsuitable, high-position video surveillance requires dispatching drones to the site to collect video, compensating for the limitations of fixed-position cameras. Thus, a target linkage tracking method and system between drones and high-position PTZ cameras is essential.

[0005] However, traditional target linkage tracking has the following drawbacks:

[0006] (1) Existing image target recognition technology adopts the latest YOLO model version (v11) technology, and trains specific targets in advance on high-resolution video and UAV video to form an experienced target recognition inference key algorithm weight file;

[0007] (2) The existing high-position video surveillance technology platform can realize the video display of many high-position video surveillance, manually operate the pan-tilt of the designated camera to turn to the target area, or operate the high-position PTZ camera to turn according to the preset monitoring and patrol trajectory.

[0008] (3) Existing UAV flight control platforms can manually generate flight missions based on the target location and dispatch UAVs from suitable airports to the target location area. This is a passive manual mode, which requires experienced and well-trained UAV dispatchers to be competent. Summary of the Invention

[0009] The purpose of this invention is to provide a target linkage tracking system and method based on UAV and high-position PTZ camera video, to solve the problems mentioned in the background art regarding existing image target recognition technologies. This invention employs the latest YOLO model version (v11) and pre-trains on high-position video and UAV video for specific targets, forming an experienced target recognition inference key algorithm weight file. Existing high-position video surveillance technology platforms display video from numerous high-position surveillance cameras, requiring manual operation of the gimbal of a designated camera to turn towards the target area, or operation of the high-position PTZ camera to turn according to a preset monitoring and patrol trajectory. Existing UAV flight control platforms manually generate flight tasks based on the target location and schedule UAVs from suitable airports to the target location area, which is a passive, manual mode requiring experienced and trained UAV dispatchers.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a target linkage tracking system based on UAV and high-position PTZ camera video, comprising a target linkage tracking system, wherein the target linkage tracking system includes a front-end device, an intermediate processing module and a back-end computing module, wherein the front-end device is connected to the intermediate processing module and the intermediate processing module is connected to the back-end computing module.

[0011] As a preferred embodiment of the present invention, the front-end equipment includes a video conferencing platform, a high-position video monitor, a drone, and a high-position PTZ camera.

[0012] As a preferred embodiment of the present invention, the intermediate processing module includes a video target recognition submodule, a PTZ processing submodule, a GPS processing submodule, a UAV information receiving submodule, a UAV mission processing submodule, an MQTT information processing submodule, and a deep neural network submodule. The video target recognition submodule, the PTZ processing submodule, and the GPS processing submodule are all connected to the high-position PTZ camera, and the UAV information receiving submodule, the UAV mission processing submodule, and the MQTT information processing submodule are all connected to the UAV.

[0013] As a preferred embodiment of the present invention, the back-end calculation module includes a coordinate calculation submodule, a target recognition submodule, a video linkage control submodule, and a video linkage analysis submodule.

[0014] This invention discloses a method for using a target linkage tracking system based on UAV and high-position PTZ camera video, comprising the following steps:

[0015] Step 1: Initialize the operating environment: Import the high-position video surveillance resource table, drone hangar table, event image recognition model library, and high-position PTZ initial parameter table from the VisionLink platform;

[0016] Step 2, Video Target Recognition: The high-position PTZ camera loads the target event image target detection model through the video target recognition submodule, determines the target's position in the display area based on the inference results, calculates the PTZ parameters that the PTZ camera should operate on in the next moment according to the preset rules, and passes the parameters to the PTZ processing submodule through the message queue. The functions of the above modules will be executed in a loop until the task is terminated.

[0017] Step 3, PTZ processing: The PTZ processing submodule cyclically receives PTZ parameter sets from other modules, compares them with the initial PTZ settings of the corresponding high-position PTZ camera, and issues horizontal, vertical and lens focal length adjustment operations for the PTZ camera. After completing the operations, it transmits the PTZ parameters after locking the target to the next GPS coordinate.

[0018] Step 4, First GPS Processing: The GPS processing submodule cyclically receives the PTZ parameter set from the PTZ processing submodule after locking the target area. It combines the GPS latitude and longitude and relative ground height h1 of the corresponding high-position PTZ camera to determine the current center location of the high-position PTZ camera lens and its distance from the high-position PTZ camera. It calculates the GPS coordinates of the target area and transmits them, along with the corresponding PTZ camera parameters and image inference event results, to the UAV flight control platform.

[0019] Step 5, Information Processing: The UAV flight control platform continuously receives the image inference results from each PTZ camera and the GPS parameters of the target's location G3. Based on the event response processing level and the association between GPS and the nearest UAV resources, it generates a UAV flight scheduling route task sheet and sends a task confirmation form to the UAV platform duty personnel.

[0020] Step Six, Second GPS Processing: After personnel confirm that the mission drone has taken off along the flight path, the MQTT information processing submodule continuously receives relevant parameters from the drone's video during flight, including the lens PTZ and the aircraft's GPS latitude and longitude G2 and altitude h2. It calculates the GPS coordinates of the center area of ​​the drone's video field of view at that time and transmits these parameters to the high-position PTZ camera's GPS receiving and processing module through a message queue for coordinated use in tracking the same target.

[0021] As a preferred technical solution of the present invention, step one involves initializing the operating environment to prepare necessary basic data for normal system operation and algorithm implementation. This includes: obtaining a high-position monitoring video resource table from the video conferencing platform, which is a directory involving all high-position monitoring and includes the device identifier, device backend login account, login password, network IP, network port, device calibration latitude and longitude, device calibration initial deflection angle, and device viewing angle for each monitoring channel; obtaining a hangar table from the UAV flight control platform, which includes the hangar IP, device serial number (SN), hangar transmission video stream address, and hangar calibration latitude and longitude; obtaining an event image recognition model library, which includes pre-trained deep neural network image recognition model files and the path information of the model files stored on the disk; obtaining a high-speed PTZ initial parameter table, logging into the PTZ camera through a program to obtain the range of values ​​for the PTZ camera's horizontal rotation angle P, vertical pitch angle T, and zoom scaling factor Z, and initializing the PTZ camera rotation to the calibration initial deflection angle.

[0022] As a preferred technical solution of the present invention, step two employs a deep neural network image recognition algorithm. This algorithm processes and analyzes the image and video data of the target captured and recorded in real-time by the high-position PTZ camera, extracts target behavioral features, identifies the target's motion trajectory and location information, and transmits the information to the PTZ processing submodule via a message queue. The deep neural network image recognition algorithm is based on the YOLO model. The MakeSense tool is used to label and classify the collected videos and images. The YOLO model is then used to train the labeled images and videos, integrates target features, and optimizes the model using geometric transformation and hybrid enhancement techniques. The optimized model is evaluated using a test dataset, and the optimal model is selected for system integration, thereby achieving target recognition from high-position PTZ camera video captures.

[0023] As a preferred embodiment of the present invention, in step three, the PTZ value of the high-position PTZ camera is automatically adjusted based on the target detection border information subscribed from the message queue, controlling the high-position PTZ camera to track the target and always placing the target in the center of the image. If a single target is input from the target detection border information, the center information (x, y) of that target detection border is taken. If multiple targets are input, the center information weighted by the confidence rate (Conf) is taken as the overall center information (x, y). In this case, the center information (x, y) is calculated by the following formula:

[0024]

[0025] ,

[0026] After calculating the center information, the program controls the high-position PTZ camera to move the target center as close as possible to the center of the screen, i.e., x=0.5 and y=0.5. In practice, to prevent the PTZ camera from shaking continuously when the target center is moved to the center of the screen and failing to lock onto the target, the aforementioned center of the screen is actually the central area, defined as: x between 0.4 and 0.6, y between 0.4 and 0.6. After the target moves to the central area of ​​the screen, a locking event is triggered after a set time, here set to 2 seconds. The horizontal rotation angle P, vertical pitch angle T, and zoom factor Z of the high-position PTZ camera at this time are then transmitted to the GPS processing submodule through a message queue.

[0027] As a preferred technical solution of the present invention, in step four, after the GPS processing submodule subscribes to the PTZ value of the high-position PTZ camera locking the target from the message queue, it combines the calibrated GPS latitude and longitude and the relative ground height h1 of the corresponding high-position PTZ camera to determine the current orientation of the center area of ​​the high-position PTZ camera lens, the distance a from the high-position PTZ camera, and calculates the GPS coordinates of the target area. The vertical rotation angle A and the horizontal rotation angle B should be the difference between the actual vertical rotation angle A1, the actual horizontal rotation angle B1 and the initial vertical offset angle A2 and the initial horizontal offset angle B2 calibrated by the high-position PTZ camera, that is, determined by the following formula:

[0028]

[0029]

[0030] When installing a high-position PTZ camera, the horizontal rotation angle is positive in the clockwise direction from north to east and negative in the counterclockwise direction; the vertical rotation angle is positive in the downward direction and negative in the upward direction. Based on the vertical rotation angle A, and according to the theorem of equal alternate angles in parallel lines and the definition of trigonometric functions, the distance 'a' between the projection of the high-position PTZ camera perpendicular to the ground and the target area on the ground is obtained by the following formula:

[0031]

[0032] After calculating the distance 'a', the projections of 'a' in the east and north directions are obtained by trigonometric functions of the horizontal deflection angle 'B', as shown in the following formula:

[0033]

[0034]

[0035] Δx and Δy represent the offset distances between the target area and the calibration coordinates of the high-position PTZ camera (latitude and longitude). The coordinates of the target area are determined by the sum of the calibration coordinates and the offset distance. The conversion relationship between Δx and Δy offsets and coordinates is as follows: for every 0.00001 degrees of longitude (Δx), the distance differs by 1 meter; for every 0.00001 degrees of latitude (Δy), the distance differs by 1.1 meters. Assuming the calibration coordinates of the high-position PTZ camera are (G1, G2), the coordinates of the target area (G3, G4) are given by the following formula:

[0036]

[0037]

[0038] The GPS coordinates of the target area are calculated and transmitted to the UAV flight control platform along with the corresponding PTZ camera parameters and image inference event results.

[0039] As a preferred embodiment of the present invention, in step five, the UAV flight control platform receives the image inference results from each PTZ camera and the GPS parameters of the target's location. Based on the event response processing level and the association between GPS and the nearest UAV resources, it generates a UAV flight scheduling route task sheet and issues a task confirmation form to the UAV platform duty personnel. Furthermore, the distance is determined using Eulerian distance, which is determined by the following formula:

[0040]

[0041] Where d is the straight-line distance, x2-x1 is the distance along the X-axis, and y2-y1 is the distance along the Y-axis.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] 1. Use a high-position PTZ camera to capture video of the target area, perform target recognition and analysis based on the acquired video information, and simultaneously coordinate with drones to collect information on-site, thus compensating for the limited field of view of fixed-position cameras;

[0044] 2. By using high-position PTZ cameras to monitor the designated area for extended periods without interruption, the limitations of drones' limited flight time and inability to continuously hover over targets to provide information to the backend are compensated. Furthermore, the automated flight path planning allows the drones to be flown as long as the personnel meet the requirements, thus solving the problem that drone video shooting requires the reliance on well-trained drone pilots.

[0045] 3. By analyzing the high-position PTZ camera video with artificial intelligence image recognition algorithms, the high-position PTZ camera is automatically adjusted so that the identified target is placed in the center area of ​​the screen. The latitude and longitude of the target are calculated by combining the real-time deflection parameters of the PTZ camera with the latitude and longitude coordinates calibrated by the PTZ camera. A drone is then dispatched to the site to supplement the video by shooting from multiple angles. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the target linkage tracking system of the present invention;

[0047] Figure 2 This is a schematic diagram of the architecture of the front-end device of the present invention;

[0048] Figure 3 This is a schematic diagram of the architecture of the intermediate processing module of the present invention;

[0049] Figure 4 This is a schematic diagram of the architecture of the backend computing module of the present invention;

[0050] Figure 5 This is a flowchart of the present invention.

[0051] In the diagram: 1. Target linkage tracking system; 2. Front-end equipment; 21. Vision linkage platform; 22. High-position video monitor; 23. UAV; 24. High-position PTZ camera; 3. Intermediate processing module; 31. Video target recognition submodule; 32. PTZ processing submodule; 33. GPS processing submodule; 34. UAV information receiving submodule; 35. UAV mission processing submodule; 36. MQTT information processing submodule; 37. Deep neural network submodule; 4. Back-end computing module; 41. Coordinate calculation submodule; 42. Target recognition submodule; 43. Video linkage control submodule; 44. Video linkage analysis submodule. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0053] Please see Figure 1-5The present invention provides a target linkage tracking system based on UAV and high-position PTZ camera video, including a target linkage tracking system 1. The target linkage tracking system 1 includes a front-end device 2, an intermediate processing module 3 and a back-end computing module 4. The front-end device 2 is connected to the intermediate processing module 3 and the intermediate processing module 3 is connected to the back-end computing module 4.

[0054] The front-end equipment 2 includes a video conferencing platform 21, a high-position video monitor 22, a drone 23, and a high-position PTZ camera 24.

[0055] The intermediate processing module 3 includes a video target recognition submodule 31, a PTZ processing submodule 32, a GPS processing submodule 33, a UAV information receiving submodule 34, a UAV mission processing submodule 35, an MQTT information processing submodule 36, and a deep neural network submodule 37. The video target recognition submodule 31, the PTZ processing submodule 32, and the GPS processing submodule 33 are all connected to the high-position PTZ camera 24, and the UAV information receiving submodule 34, the UAV mission processing submodule 35, and the MQTT information processing submodule 36 are all connected to the UAV 23.

[0056] The backend calculation module 4 includes a coordinate calculation submodule 41, a target recognition submodule 42, a video linkage control submodule 43, and a video linkage analysis submodule 44.

[0057] This invention discloses a method for using a target linkage tracking system based on UAV and high-position PTZ camera video, comprising the following steps:

[0058] Step 1: Initialize the operating environment: Import the high-position video surveillance resource table, UAV hangar table, event image recognition model library, and high-position PTZ initial parameter table from the VisionLink platform 21;

[0059] Step 2, Video Target Recognition: The high-position PTZ camera 24 loads the target event image target detection model through the video target recognition submodule 31, determines the target's position in the display area according to the inference results, calculates the PTZ parameters that the PTZ camera should operate at the next moment according to the preset rules, and transmits the parameters to the PTZ processing submodule 32 through the message queue. The above module functions will be executed in a loop until the task is terminated.

[0060] Step 3, PTZ processing: The PTZ processing submodule 32 cyclically receives PTZ parameter sets from other modules, compares them with the initial PTZ settings of the corresponding high-position PTZ camera 24, and issues horizontal, vertical and lens focal length adjustment operations for the PTZ camera. After completing the operations, it transmits the PTZ parameters after locking the target to the next GPS coordinate.

[0061] Step 4, First GPS Processing: The GPS processing submodule 33 cyclically receives the PTZ parameter group from the PTZ processing submodule 32 after locking the target area. It combines the GPS latitude and longitude and relative ground height h1 of the corresponding high-position PTZ camera 24, determines the current center area of ​​the high-position PTZ camera 24, the distance from the high-position PTZ camera 24, calculates the GPS coordinates of the target area, and transmits the corresponding PTZ camera parameters and image inference event results to the UAV flight control platform.

[0062] Step 5, Information Processing: The UAV flight control platform continuously receives the image inference results from each PTZ camera and the GPS parameters of the target's location G3. Based on the event response processing level and the association between GPS and the nearest UAV resources, it generates a UAV flight scheduling route task sheet and sends a task confirmation form to the UAV platform duty personnel.

[0063] Step Six, Second GPS Processing: After personnel confirm that the mission drone 23 has taken off according to the flight path, the MQTT information processing submodule 36 continuously receives relevant parameters of the drone video in flight, including the lens PTZ and the aircraft's GPS latitude and longitude G2 and altitude h2. It calculates the GPS coordinates of the center area of ​​the drone's video field of view at that time and transmits these parameters to the GPS receiving and processing module of the high-position PTZ camera 24 through a message queue for use in linkage tracking of the same target.

[0064] Step one initializes the runtime environment, preparing necessary basic data for normal system operation and algorithm implementation. This includes: obtaining the high-position monitoring video resource table from the VisionLink platform 21, which is a directory involving all high-position monitoring, containing the device identifier, device backend login account, login password, network IP, network port, device calibration latitude and longitude, device calibration initial deflection angle, and device viewing angle for each monitoring channel; obtaining the hangar table from the UAV flight control platform 23, which contains the hangar IP, device serial number (SN), hangar transmission video stream address, and hangar calibration latitude and longitude; obtaining the event image recognition model library, which contains pre-trained deep neural network image recognition model files and the path information of the model files stored on the disk; obtaining the high-speed PTZ initial parameter table, logging into the PTZ camera through the program to obtain the range of values ​​for the PTZ camera's horizontal rotation angle P, vertical pitch angle T, and zoom scaling factor Z, and initializing the PTZ camera rotation to the calibration initial deflection angle.

[0065] Step two employs a deep neural network image recognition algorithm. This algorithm processes and analyzes the image and video data captured and recorded in real-time by the high-position PTZ camera (24-bit), extracting target behavioral features and identifying the target's motion trajectory and location information. This information is then transmitted to the PTZ processing submodule (32) via a message queue for further processing. The deep neural network image recognition algorithm is based on the YOLO model. The MakeSense tool is used to label and classify the collected videos and images. The YOLO model is then used to train the labeled images and videos, integrating target features. Geometric transformations and hybrid enhancement techniques are used to optimize the model. The optimized model is evaluated using a test dataset, and the optimal model is selected for system integration, enabling the recognition of targets captured by the high-position PTZ camera (24-bit video).

[0066] In step three, based on the target detection bounding box information subscribed from the message queue, the PTZ value of the high-position PTZ camera 24 is automatically adjusted to control the high-position PTZ camera 24 to track the target and always keep the target in the center of the image. If a single target is input in the target detection bounding box information, the center information x,y of that target detection bounding box is taken. If multiple targets are input, the center information weighted by the confidence rate is taken as the overall center information x,y. At this time, the center information x,y is calculated by the following formula:

[0067]

[0068] ,

[0069] After calculating the center information, the program controls the high-position PTZ camera 24 to move the target center as close as possible to the center of the screen, i.e., x=0.5 and y=0.5. In practice, to prevent the PTZ camera from shaking continuously when the target center is moved to the center of the screen and thus failing to lock onto the target, the aforementioned center of the screen is actually the central area, defined as: x between 0.4 and 0.6, y between 0.4 and 0.6. After the target moves to the central area of ​​the screen, a lock event is triggered after a set time, here set to 2 seconds. The horizontal rotation angle P, vertical pitch angle T, and zoom factor Z of the high-position PTZ camera 24 at this time are then transmitted to the GPS processing submodule 33 through the message queue.

[0070] In step four, after the GPS processing submodule 33 subscribes to the PTZ value of the high-position PTZ camera 24 that has locked onto the target from the message queue, it combines the calibrated GPS latitude and longitude of the corresponding high-position PTZ camera 24 with the relative ground height h1 to determine the current location of the center area of ​​the high-position PTZ camera 24 lens and its distance from the high-position PTZ camera 24 by 'a'. It then calculates the GPS coordinates of the target area. The vertical rotation angle A and the horizontal rotation angle B should be the difference between the actual vertical rotation angle A1, the actual horizontal rotation angle B1 and the initial vertical offset angle A2 and the initial horizontal offset angle B2 calibrated by the high-position PTZ camera 24, which are determined by the following formula:

[0071]

[0072]

[0073] When the high-position PTZ camera 24 is installed, the horizontal rotation angle is positive in the clockwise direction from north to east and negative in the counterclockwise direction; the vertical rotation angle is positive in the downward direction and negative in the upward direction. Based on the vertical rotation angle A, and according to the theorem of equal alternate angles in parallel lines and the definition of trigonometric functions, the distance 'a' between the projection of the high-position PTZ camera 24 perpendicular to the ground and the target area on the ground is obtained by the following formula:

[0074]

[0075] After calculating the distance 'a', the projections of 'a' onto the east and north directions are obtained using trigonometric functions of the horizontal deflection angle 'B', as shown in the following formula:

[0076]

[0077]

[0078] Δx and Δy represent the offset distances between the target area and the calibration coordinates (latitude and longitude) of the high-position PTZ camera 24. The coordinates (latitude and longitude) of the target area are determined by the sum of the calibration coordinates (latitude and longitude) of the high-position PTZ camera 24 and the offset distance. The conversion relationship between Δx and Δy offsets and coordinates is as follows: for every 0.00001 degrees of longitude Δx, the distance difference is 1 meter; for every 0.00001 degrees of latitude Δy, the distance difference is 1.1 meters. Assuming the calibration coordinates of the high-position PTZ camera 24 are G1 and G2, then the coordinates (G3 and G4) of the target area are given by the following formulas:

[0079]

[0080]

[0081] The GPS coordinates of the target area are calculated and transmitted to the UAV flight control platform along with the corresponding PTZ camera parameters and image inference event results.

[0082] In step five, the UAV flight control platform receives the image inference results from each PTZ camera and the GPS parameters of the target's location. Based on the event response processing level and the correlation between GPS and the nearest UAV resources, it generates a UAV flight scheduling route task sheet and sends a task confirmation form to the UAV platform duty personnel. Furthermore, the distance is determined using Eulerian distance, which is calculated by the following formula:

[0083]

[0084] Where d is the straight-line distance, x2-x1 is the distance along the X-axis, and y2-y1 is the distance along the Y-axis.

[0085] In this invention, the high-position video surveillance resource table, UAV hangar table, event image recognition model library, and PTZ initial parameter table of the high-position PTZ camera 24 are imported from the video conferencing platform 21. The high-position PTZ camera 24 loads the target event image target detection model through the video target recognition submodule 31, determines the target's position in the display area according to the inference results, calculates the PTZ parameters that the camera should operate on in the next moment according to the preset rules, and transmits the parameters to the PTZ processing submodule 32 through the message queue. The functions of the above modules will be executed cyclically until the task terminates. The PTZ processing submodule 32 cyclically receives PTZ parameter sets from other modules, compares them with the initial PTZ settings of the corresponding high-position PTZ camera 24, and issues horizontal, vertical, and lens focal length adjustment operation actions for the camera. After the action is completed, the PTZ parameters after locking the target are transmitted to the next GPS coordinate; the GPS processing submodule 33 cyclically receives the PTZ parameter group after the target area is locked by the PTZ processing submodule 32, combines the GPS latitude and longitude and relative ground height h1 of the corresponding high-position PTZ camera 24, determines the current center area of ​​the high-position PTZ camera 24, the distance from the high-position PTZ camera 24, calculates the GPS coordinates of the target area, and transmits them together with the corresponding PTZ camera parameters and image inference event results to the UAV flight control platform; the UAV flight control platform cyclically receives the image inference results from each PTZ camera and the GPS parameters of the target location G3, and generates, according to the event response processing level and the association between GPS and the nearest UAV resources, generates The system generates a flight path task order for the UAV and sends a task confirmation form to the UAV platform duty personnel. After the personnel confirm that the UAV 23 has taken off according to the flight path, the MQTT information processing submodule 36 continuously receives relevant parameters of the UAV video during flight, including the lens PTZ and the aircraft's GPS latitude and longitude G2 and altitude h2. It calculates the GPS coordinates of the center area of ​​the UAV's video field of view at that time and transmits these parameters to the GPS receiving and processing module of the high-position PTZ camera 24 through a message queue for use in linkage tracking of the same target. After the duty personnel confirm that the UAV 23 has taken off according to the flight path, the UAV 23 flight control platform will continue to receive relevant parameters of the UAV video during flight, including the gimbal lens PTZ and the aircraft's GPS latitude and longitude (G2) and altitude (h2). The same calculation method is used to calculate the GPS coordinates of the target's center area in the video field of view of UAV 23 at that time, and this parameter is transmitted to the high-position PTZ camera 24 and the GPS receiving and processing module through a message queue for coordinated tracking of the same target. Utilizing the Internet of Things architecture and message queue technology, the front-end device 2, UAV signal, high-position video surveillance streaming media, coordinate calculation, target recognition, video linkage control, and video recognition analysis are integrated into the back-end computing unit. This completes the process from target discovery and recognition by the high-position PTZ camera 24 to continuous visual tracking of the target by the coordinated UAV, enabling UAV 23 to track the target and sharing the same viewpoint between UAV 23 and the high-position PTZ camera 24, thus complementing each other's advantages. This provides a more comprehensive and multi-dimensional solution for emergency monitoring and urban planning.

[0086] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for using a target linkage tracking system based on UAV and high-position PTZ camera video, comprising a target linkage tracking system (1), characterized in that: The target linkage tracking system (1) includes a front-end device (2), an intermediate processing module (3) and a back-end computing module (4). The front-end device (2) is connected to the intermediate processing module (3), and the intermediate processing module (3) is connected to the back-end computing module (4). The front-end equipment (2) includes a video conferencing platform (21), a high-position video monitor (22), a drone (23), and a high-position PTZ camera (24). The intermediate processing module (3) includes a video target recognition submodule (31), a PTZ processing submodule (32), a GPS processing submodule (33), a UAV information receiving submodule (34), a UAV mission processing submodule (35), an MQTT information processing submodule (36), and a deep neural network submodule (37). The video target recognition submodule (31), the PTZ processing submodule (32), and the GPS processing submodule (33) are all connected to the high-position PTZ camera (24), and the UAV information receiving submodule (34), the UAV mission processing submodule (35), and the MQTT information processing submodule (36) are all connected to the UAV (23). The back-end computing module (4) includes a coordinate calculation submodule (41), a target recognition submodule (42), a video linkage control submodule (43), and a video linkage analysis submodule (44). Includes the following steps: Step 1: Initialize the operating environment: Import the high-position video surveillance resource table, UAV hangar table, event image recognition model library, and high-position PTZ initial parameter table from the VisionLink platform (21); Step 2, Video Target Recognition: The high-position PTZ camera (24) loads the target event image target detection model through the video target recognition submodule (31), determines the target's position in the display area according to the reasoning results, calculates the PTZ parameters that the PTZ camera should operate at the next moment according to the pre-set rules, and passes the parameters to the PTZ processing submodule (32) through the message queue. The above module functions will be executed in a loop until the task is terminated. Step 3, PTZ processing: The PTZ processing submodule (32) cyclically receives PTZ parameter sets from other modules, compares them with the initial PTZ settings of the corresponding high-position PTZ camera (24), and issues horizontal, vertical and lens focal length adjustment operations for the PTZ camera. After completing the operations, it transmits the PTZ parameters after locking the target to the first GPS coordinate processing module. Step 4, First GPS Processing: The GPS processing submodule (33) cyclically receives the PTZ parameter group from the PTZ processing submodule (32) after locking the target area. It combines the GPS latitude and longitude and relative ground height h1 of the corresponding high-position PTZ camera (24), determines the current center area of ​​the high-position PTZ camera (24), the distance from the high-position PTZ camera (24), calculates the GPS coordinates of the target area, and transmits the corresponding PTZ camera parameters and image inference event results to the UAV flight control platform. Step 5, Information Processing: The UAV flight control platform continuously receives the image inference results from each PTZ camera and the GPS parameters of the target's location G3. Based on the event response processing level and the association between GPS and the nearest UAV resources, it generates a UAV flight scheduling route task sheet and sends a task confirmation form to the UAV platform duty personnel. Step 6, Second GPS Processing: After the event verification personnel confirm that the mission UAV (23) has taken off according to the flight path, the MQTT information processing submodule (36) continuously receives relevant parameters of the UAV video in flight, including the lens PTZ and the aircraft's GPS latitude and longitude G2 and altitude h2, calculates the GPS coordinates of the center area of ​​the UAV's video field of view at that time, and transmits this parameter to the GPS receiving and processing module of the high-position PTZ camera (24) through the message queue for use in linkage tracking of the same target.

2. The method of using a target linkage tracking system based on UAV and high-position PTZ camera video according to claim 1, characterized in that: In step one, the runtime environment is initialized to prepare the necessary basic data for the normal operation of the system and the implementation of the algorithm.

3. The method of using a target linkage tracking system based on UAV and high-position PTZ camera video according to claim 1, characterized in that: In step two, a deep neural network image recognition algorithm is used to process and analyze the image and video data of the target captured and recorded in real time by the high-position PTZ camera (24), extract the target behavior features, identify the target's motion trajectory and location information, and transmit the information to the PTZ processing submodule (32) through a message queue for processing. The deep neural network image recognition algorithm is based on the YOLO model. The MakeSense tool is used to label and classify the collected videos and images. The YOLO model is used to train the labeled images and videos, integrate the target features, optimize the model using geometric transformation and hybrid enhancement techniques, evaluate the optimized model using a test dataset, select the optimal model for system integration, and realize the target recognition of the image captured by the high-position PTZ camera (24).

4. The method of using a target linkage tracking system based on UAV and high-position PTZ camera video according to claim 1, characterized in that: In step three, the PTZ value of the high-position PTZ camera (24) is automatically adjusted based on the target detection bounding box information subscribed from the message queue. The high-position PTZ camera (24) is controlled to track the target and always keep the target in the center of the screen. If a single target is entered in the target detection bounding box information, the center information (x, y) of that target detection bounding box is taken. If multiple targets are entered, the center information weighted by the confidence rate (Conf) is taken as the overall center information (x, y). At this time, the center information (x, y) is calculated by the following formula: After calculating the center information, the program controls the high-position PTZ camera (24) to move the target center as close as possible to the center of the screen, i.e., x=0.5 and y=0.

5. The aforementioned center of the screen is actually the central area, defined as: x is between 0.4 and 0.6, and y is between 0.4 and 0.

6. After the target moves to the center area of ​​the screen, it continues for a set time, which is set to 2 seconds here, to trigger the lock event. The horizontal rotation angle P, vertical pitch angle T, and zoom factor Z of the high-position PTZ camera (24) at this time are transmitted to the GPS processing submodule (33) through the message queue.

5. The method of using a target linkage tracking system based on UAV and high-position PTZ camera video according to claim 1, characterized in that: In step four, after the GPS processing submodule (33) subscribes to the PTZ value of the high-position PTZ camera (24) that has locked the target from the message queue, it combines the calibrated GPS latitude and longitude and relative ground height h1 of the corresponding high-position PTZ camera (24) to determine the current location of the center area of ​​the high-position PTZ camera (24) lens and its distance from the high-position PTZ camera (24) by a, and calculates the GPS coordinates of the target area. The vertical rotation angle A and the horizontal rotation angle B should be the difference between the actual vertical rotation angle A1, the actual horizontal rotation angle B1 and the initial vertical offset angle A2 and the initial horizontal offset angle B2 calibrated by the high-position PTZ camera (24), that is, determined by the following formula: A = A 1 - A 2 B = B 1 - B 2 When the high-position PTZ camera (24) is installed, the horizontal rotation angle is positive in the clockwise direction from north to east and negative in the counterclockwise direction; the vertical rotation angle is positive in the downward direction and negative in the upward direction. According to the vertical rotation angle A, and based on the theorem of equal alternate angles in parallel lines and the definition of trigonometric functions, the distance a between the projection of the high-position PTZ camera (24) perpendicular to the ground and the target area on the ground is obtained by the following formula: After calculating the distance 'a', the projections of 'a' in the east and north directions are obtained by trigonometric functions of the horizontal deflection angle 'B', as shown in the following formula: D x = ax· sin B D y = ax · cos B Δx and Δy are the offset distances of the detection target area from the calibration coordinates of the high-position PTZ camera (24). The coordinates of the detection target area are determined by the sum of the calibration coordinates of the high-position PTZ camera (24) and the offset distance. The conversion relationship between the offsets of Δx and Δy and the coordinates is as follows: for every 0.00001 degrees of longitude Δx, the distance difference is 1 meter; for every 0.00001 degrees of latitude Δy, the distance difference is 1.1 meters. Assuming that the calibration coordinates of the high-position PTZ camera (24) are (G1, G2), the coordinates of the detection target area (G3, G4) are given by the following formula: The GPS coordinates of the target area are calculated and transmitted to the UAV flight control platform along with the corresponding PTZ camera parameters and image inference event results.

6. The method of using a target linkage tracking system based on UAV and high-position PTZ camera video according to claim 1, characterized in that: In step five, the UAV flight control platform receives the image inference results from each PTZ camera and the GPS parameters of the target's location. Based on the event response processing level and the association between GPS and the nearest UAV resources, it generates a UAV flight scheduling route task sheet and issues a task confirmation form to the UAV platform's on-duty event verification personnel. Furthermore, the distance is determined using Eulerian distance, which is calculated by the following formula: d = Where d is the straight-line distance, x2-x1 is the distance along the X-axis, and y2-y1 is the distance along the Y-axis.

Images