Target linkage tracking system and method based on videos of unmanned aerial vehicle and high-position dome camera

Abstract

The invention discloses a target linkage tracking system and method based on videos of an unmanned aerial vehicle and a high-position dome camera, and the system comprises a target linkage tracking system, the target linkage tracking system comprises a front device, an intermediate processing module and a rear-end calculation module, the front device is connected with the intermediate processing module, the intermediate processing module is connected with the rear-end calculation module, and the rear-end calculation module is connected with the rear-end calculation module. According to the invention, a high-position dome camera is used to capture a video of a target area, target identification analysis is carried out through collected video information, and an unmanned aerial vehicle is linked and dispatched to a site to collect information, thereby making up for insufficient angle view of a camera at a fixed position; the defects that the endurance time of the unmanned aerial vehicle is limited and the unmanned aerial vehicle cannot continuously stay over a target to provide information for the rear end are overcome through long-time and uninterrupted monitoring of a set area by the high-position dome camera, a route is automatically planned, the unmanned aerial vehicle can fly as long as personnel accord with the situation, and the problem that the unmanned aerial vehicle needs to depend on training a privileged unmanned aerial vehicle pilot during video shooting is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) and high-position ball camera video, and in particular to a target linkage tracking system and method based on unmanned aerial vehicle (UAV) and high-position ball camera video. Background Art

[0002] Compared with low-position video surveillance at a height of less than 20 meters, the advantage of high-position ball camera video surveillance is its wide field of view, and the patrol area is an exponential expansion of the low-position video. A high-speed, high-definition camera ball camera is installed at a high position, which can patrol the surrounding site conditions at a fixed position 24 hours a day. The emergence of drones, the field of view of the cameras they carry, makes up for the shortcoming of insufficient target acquisition angles of high-position video surveillance. They can fly over the target and conduct video surveillance of the target area from all angles without blind spots to obtain the target details required on site. However, the shortcoming of drones is that their flight time is limited, and they cannot continue to stay over the target to provide information to the back end. The linkage of drones and high-position video surveillance technologies can achieve effective video monitoring of a large area with a combination of static and dynamic and complementary advantages.

[0003] China Tower Corporation Limited (hereinafter referred to as "Tower Company") owns the largest number of communication tower resources in the country. It is common to use the high-position tower resources to install high-speed and high-definition ball cameras for application in various industries. They are widely deployed across the country in the fields of forest and grassland fire prevention, river basin protection, road traffic security, and farmland protection, and have played a positive role. With the further development of the informationization of the integrated air-ground-space perception system and the rapid development of the low-altitude economy, according to actual needs, the "Sky Eye" linkage method of realizing high-position monitoring and drone monitoring in the field of video surveillance will become one of the key work requirements for the development of high-quality video surveillance information systems in various industries.

[0004] High-position video surveillance is generally positioned so that the camera lens is more than 20 meters above the ground, while the height of high-speed ball cameras installed on communication towers is generally around 40 meters; the height of drone video is generally between 80 and 150 meters. Engineering experience data shows that the farthest distance of effective target recognition field of view pixels is generally within 10 times the lens height, that is, a high-position ball camera with a height of 40 meters has a maximum ground field of view of more than 2,000 meters, while the effective range of video screen pixels is about 400 meters; and the maximum field of view of drone video at a flight altitude of 120 meters can exceed 6,000 meters, and the effective range of video screen pixels is about 1,200 meters. Therefore, when the distance exceeds a certain distance and the observation angle is not suitable, high-position video surveillance needs to dispatch drones to the scene to collect videos to make up for the shortcomings of fixed-position cameras. Therefore, it is very necessary to realize a method and system for target linkage tracking between drones and high-position ball camera videos.

[0005] However, traditional target linkage tracking has the following disadvantages:

[0006] (1) The existing image target recognition technology uses the latest YOLO model version (v11) technology, which is trained on specific targets in advance using high-position videos and drone videos to form an experienced target recognition reasoning key algorithm weight file;

[0007] (2) The existing high-position video surveillance technology platform can realize the video display of many high-position video surveillance, manually operate the pan / tilt of a designated camera to turn to the target area, or operate the high-position ball camera to turn according to the preset monitoring patrol track;

[0008] (3) The existing UAV flight control platform can manually generate flight missions based on the target location and dispatch UAVs at suitable airports to the target location area. This is a passive manual mode that requires experienced and well-trained UAV dispatchers to perform. Summary of the invention

[0009] The purpose of the present invention is to provide a target linkage tracking system and method based on unmanned aerial vehicle (UAV) and high-position ball camera videos, so as to solve the existing image target recognition technology proposed in the above background technology, and adopt the latest YOLO model version (v11) technology, by pre-training high-position video and UAV video for specific targets, to form an experienced target recognition reasoning key algorithm weight file; the existing high-position video monitoring technology platform realizes the video display of many high-position video monitoring, manually operates the pan-tilt of the designated camera to turn to the target area, or operates the high-position ball camera camera to turn according to the preset monitoring patrol track; the existing UAV flight control platform manually generates flight missions according to the target position, and dispatches the UAV of the appropriate airport to the target position area, which belongs to the passive manual mode and requires experienced and well-trained UAV dispatchers to be competent.

[0010] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a target linkage tracking system based on unmanned aerial vehicle and high-position ball camera video, including a target linkage tracking system, the target linkage tracking system includes a front-end device, an intermediate processing module and a back-end computing module, the front-end device is connected to the intermediate processing module, and the intermediate processing module is connected to the back-end computing module.

[0011] As a preferred technical solution of the present invention, the front-end equipment includes a visual connection platform, a high-position video monitor, a drone and a high-position ball camera.

[0012] As a preferred technical solution of the present invention, the intermediate processing module includes a video target recognition submodule, a PTZ processing submodule, a GPS processing submodule, a drone information receiving submodule, a drone task processing submodule, an MQTT information processing submodule and a deep neural network submodule. The video target recognition submodule, the PTZ processing submodule and the GPS processing submodule are all connected to the high-position ball camera, and the drone information receiving submodule, the drone task processing submodule and the MQTT information processing submodule are all connected to the drone.

[0013] As a preferred technical solution of the present invention, the back-end calculation module includes a coordinate calculation submodule, a target recognition submodule, a video linkage control submodule and a video linkage analysis submodule.

[0014] The present invention provides a method for using a target linkage tracking system based on a drone and a high-position ball camera video, comprising the following steps:

[0015] Step 1: Initialize the operating environment: import the high-position video surveillance resource table, drone hangar table, event image recognition model library, and high-position ball camera PTZ initial parameter table from the video link platform;

[0016] Step 2: Video target recognition: The high-position ball camera loads the target event image target detection model through the video target recognition submodule, determines the target position in the display area according to the inference result, calculates the PTZ parameters that the ball camera should operate at the next moment according to the pre-set rules, and passes the parameters to the PTZ processing submodule through the message queue. The above module functions will be executed in a loop until the task is terminated;

[0017] Step 3, PTZ processing: The PTZ processing submodule cyclically receives the PTZ parameter group sent from other modules, compares it with the initial PTZ setting of the corresponding high-position ball camera, issues the horizontal, vertical and lens focal length adjustment operation of the ball camera, and after completing the operation, transmits the PTZ parameters after locking the target to the next GPS coordinate;

[0018] Step 4, first GPS processing: The GPS processing submodule cyclically receives the PTZ parameter group after the PTZ processing submodule locks the target area, collects the GPS longitude and latitude of the corresponding high-position ball camera and the relative ground height h1, determines the current position of the center area of ​​the high-position ball camera lens, the distance from the high-position ball camera, calculates the GPS coordinates of the target area, and transmits the corresponding ball camera parameters and image reasoning event results to the UAV flight control platform;

[0019] Step 5: Information processing: The UAV flight control platform cyclically receives the image inference results from each ball camera and the GPS parameters of the target location G3, generates a UAV flight dispatch route task list based on the event response processing level and the association between GPS and the nearest UAV resources, and issues a task confirmation list to the UAV platform duty personnel;

[0020] Step 6. Second GPS processing: After the personnel confirm that the mission drone takes off according to the route, the MQTT information processing submodule continues to receive the relevant parameters of the drone video in flight, the lens PTZ and the aircraft GPS longitude and latitude G2 and altitude h2, calculates the GPS coordinates of the center area of ​​the drone video field of view at that time, and transmits this parameter to the high-position ball camera GPS receiving and processing module through the message queue for linkage tracking of the same target.

[0021] As a preferred technical solution of the present invention, in the step one, the operating environment is initialized to prepare the necessary basic data for the normal operation of the system and the implementation of the algorithm, including: obtaining the high-position monitoring video resource table in the visual connection platform, which is a directory involving all high-position monitoring, including the device identification of each monitoring channel, the device background login account, login password, network IP, network port, device calibrated longitude and latitude, device calibrated initial deflection angle, and device viewing angle; obtaining the hangar table in the UAV flight control platform, which includes the hangar IP, device serial number SN code, hangar transmission video stream address, and hangar calibrated longitude and latitude; obtaining the event image recognition model library, including the model file of the trained deep neural network image recognition and the path information of the model file stored in the disk; obtaining the high-speed ball camera PTZ initial parameter table, logging in to the ball camera through the program to obtain the value range of the ball camera's horizontal rotation angle P, vertical pitch angle T, and zoom magnification Z, and initialize the ball camera to rotate to the calibrated initial deflection angle.

[0022] As a preferred technical solution of the present invention, a deep neural network image recognition algorithm is used in the step 2. The image and video data of the target captured and recorded in real time by the high-position ball camera video are processed and analyzed, the target behavior characteristics are extracted, the motion trajectory and position information of the target are identified, and the information is transmitted to the PTZ processing submodule through a message queue for processing. The deep neural network image recognition algorithm is based on the YOLO model, and the collected videos and pictures are labeled and classified using the Make Sense tool. The labeled pictures and videos are trained using the YOLO model, the target features are integrated, the model is optimized using geometric transformation and hybrid enhancement technology, the optimized model is evaluated using a test data set, and the optimal model is selected for system integration to achieve image recognition targets captured by the high-position ball camera video.

[0023] As a preferred technical solution of the present invention, in step 3, the PTZ value of the high-position ball camera is automatically adjusted according to the target detection frame information subscribed from the message queue, and the high-position ball camera is controlled to track the target, and the target is always placed in the center of the screen. If a single recognition target is passed in the read target detection frame information, the center information (x, y) of the target detection frame is taken. If multiple recognition targets are passed in, the center information weighted by the confidence rate is taken as the overall center information (x, y). At this time, the center information (x, y) is calculated by the following formula:

[0024]

[0025] After calculating the center information, the program controls the high-position ball camera to move the target center as close to the center of the picture as possible, that is, x=0.5 and y=0.5. In practice, in order to prevent the ball camera from shaking constantly when the target center moves to the center of the picture and cannot lock the target, the aforementioned picture center is actually the central area, defined as: x is between 0.4 and 0.6, y is between 0.4 and 0.6. When the target moves to the center area of ​​the picture and lasts for a set time, which is set to 2 seconds here, the locking event is triggered, and the horizontal rotation angle P, vertical pitch angle T, and zoom factor Z of the high-position ball camera at this time are transmitted to the GPS processing submodule through the message queue.

[0026] As a preferred technical solution of the present invention, after the GPS processing submodule in step 4 subscribes to the PTZ value of the high-position ball camera that locks the target from the message queue, it combines the calibrated GPS longitude and latitude of the corresponding high-position ball camera and the relative ground height h1 to determine the current position of the center area of ​​the high-position ball camera lens and the distance a from the high-position ball camera, and calculates the GPS coordinates of the target area. The vertical angle A and the horizontal angle B should be the difference between the actual vertical angle A1, the actual horizontal angle B1 and the initial vertical offset angle A2 and the initial horizontal offset angle B2 calibrated by the high-position ball camera, that is, determined by the following formula:

[0027] A=A1-A2

[0028] B=B1-B2

[0029] Among them, when the high-position ball camera is installed, the horizontal rotation angle is positive from north to east clockwise and negative from counterclockwise; the vertical rotation angle is positive from horizontal downward and negative from upward. According to the vertical rotation angle A, the equal theorem of alternate angles in parallel lines and the definition of trigonometric functions, the distance a between the projection of the high-position ball camera vertical to the ground and the ground target area is obtained by the following formula:

[0030]

[0031] After calculating the distance a, the projection of a in the due east and due north directions is calculated by the trigonometric function of the horizontal deflection angle B, as follows:

[0032] Δx=axsinB

[0033] Δy=axcosB

[0034] Δx and Δy are the offset distances between the longitude and latitude of the detection target area and the calibration coordinates of the high-position ball camera. The longitude and latitude of the detection target area are determined by the sum of the longitude and latitude of the calibration coordinates of the high-position ball camera and the offset distance. The conversion relationship between the Δx and Δy offsets and the coordinates is that the longitude (Δx) differs by 1 meter for every 0.00001 degree, and the latitude (Δy) differs by 1.1 meters for every 0.00001 degree. Assuming that the calibration coordinates of the high-position ball camera are (G1, G2), the coordinates of the detection target area (G3, G4) are given by the following formula:

[0035]

[0036] G4=G2+ΔxX0.00001

[0037] The GPS coordinates of the target area are calculated and transmitted to the UAV flight control platform together with the corresponding ball camera parameters and image reasoning event results.

[0038] As a preferred technical solution of the present invention, in step 5, the UAV flight control platform receives the image inference results from each ball camera and the GPS parameters of the target location, generates a UAV flight scheduling route task list according to the event response processing level and the association between GPS and the nearest UAV resource, and issues a task confirmation list to the UAV platform duty personnel. Further, the Euler distance is used to determine the distance, which is determined by the following formula:

[0039]

[0040] Where d is the straight-line distance, x2-x1 is the distance in the X-axis direction, and y2-y1 is the distance in the Y-axis direction.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. Use a high-position ball camera to capture the video of the target area, perform target recognition analysis based on the collected video information, and simultaneously dispatch drones to the scene to collect information, making up for the lack of angle field of view of fixed-position cameras;

[0043] 2. The high-position ball camera can monitor the set area for a long time and continuously to make up for the limited flight time of the drone, which cannot stay over the target continuously to provide information to the back-end. The route is automatically planned, and only personnel who meet the conditions can fly, solving the problem that drone video shooting requires well-trained drone pilots;

[0044] 3. By analyzing the video from the high-position ball camera with an artificial intelligence image recognition algorithm, the high-position ball camera is automatically adjusted to place the identified target in the center of the picture. The longitude and latitude of the target are calculated using the real-time deflection parameters of the ball camera combined with the longitude and latitude coordinates calibrated by the ball camera, and a drone is jointly dispatched to the scene to shoot video supplements from multiple angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the architecture of the target linkage tracking system of the present invention;

[0046] Figure 2 It is a schematic diagram of the architecture of the front-end device of the present invention;

[0047] Figure 3 It is a schematic diagram of the architecture of the intermediate processing module of the present invention;

[0048] Figure 4 It is a schematic diagram of the architecture of the back-end computing module of the present invention;

[0049] Figure 5 It is a flow chart of the present invention.

[0050] In the figure: 1. Target linkage tracking system; 2. Front-end equipment; 21. Visual connection platform; 22. High-position video monitor; 23. UAV; 24. High-position ball camera; 3. Intermediate processing module; 31. Video target recognition submodule; 32. PTZ processing submodule; 33. GPS processing submodule; 34. UAV information receiving submodule; 35. UAV task processing submodule; 36. MQTT information processing submodule; 37. Deep neural network submodule; 4. Back-end computing module; 41. Coordinate calculation submodule; 42. Target recognition submodule; 43. Video linkage control submodule; 44. Video linkage analysis submodule. DETAILED DESCRIPTION

[0051] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0052] See also Figure 1-5 The present invention provides a target linkage tracking system based on unmanned aerial vehicle and high-position ball camera video, including a target linkage tracking system 1, the target linkage tracking system 1 includes a front-end device 2, an intermediate processing module 3 and a back-end computing module 4, the front-end device 2 is connected to the intermediate processing module 3, and the intermediate processing module 3 is connected to the back-end computing module 4.

[0053] The front-end equipment 2 includes a video connection platform 21, a high-position video monitor 22, a drone 23 and a high-position ball camera 24.

[0054] The intermediate processing module 3 includes a video target recognition submodule 31, a PTZ processing submodule 32, a GPS processing submodule 33, a drone information receiving submodule 34, a drone task processing submodule 35, an MQTT information processing submodule 36 and a deep neural network submodule 37. The video target recognition submodule 31, the PTZ processing submodule 32 and the GPS processing submodule 33 are all connected to the high-position ball camera 24, and the drone information receiving submodule 34, the drone task processing submodule 35 and the MQTT information processing submodule 36 are all connected to the drone 23.

[0055] The back-end calculation module 4 includes a coordinate calculation submodule 41 , a target recognition submodule 42 , a video linkage control submodule 43 and a video linkage analysis submodule 44 .

[0056] The present invention provides a method for using a target linkage tracking system based on a drone and a high-position ball camera video, comprising the following steps:

[0057] Step 1: Initialize the operating environment: import the high-position video surveillance resource table, drone hangar table, event image recognition model library, and high-position ball camera 24PTZ initial parameter table from the VisionLink platform 21;

[0058] Step 2, video target recognition: The high-position ball camera 24 loads the target event image target detection model through the video target recognition submodule 31, determines the position of the target in the display area according to the inference result, calculates the PTZ parameters that the ball camera should operate at the next moment according to the pre-set rules, and passes the parameters to the PTZ processing submodule 32 through the message queue. The above module functions will be executed in a loop until the task is terminated;

[0059] Step 3, PTZ processing: The PTZ processing submodule 32 cyclically receives the PTZ parameter group sent from other modules, compares it with the initial PTZ setting of the corresponding high-position ball camera 24, issues the horizontal, vertical and lens focal length adjustment operation of the ball camera, and transmits the PTZ parameters after locking the target to the next GPS coordinate after completing the operation;

[0060] Step 4, first GPS processing: The GPS processing submodule 33 cyclically receives the PTZ parameter group after the PTZ processing submodule 32 locks the target area, collects the GPS longitude and latitude and relative ground height h1 of the corresponding high-position ball camera 24, determines the current position of the center area of ​​the high-position ball camera 24 lens, the distance from the high-position ball camera 24, calculates the GPS coordinates of the target area, and transmits the corresponding ball camera parameters and image reasoning event results to the UAV flight control platform;

[0061] Step 5: Information processing: The UAV flight control platform cyclically receives the image inference results from each ball camera and the GPS parameters of the target location G3, generates a UAV flight dispatch route task list based on the event response processing level and the association between GPS and the nearest UAV resources, and issues a task confirmation list to the UAV platform duty personnel;

[0062] Step 6, second GPS processing: After the personnel confirms that the mission drone 23 takes off according to the route, the MQTT information processing submodule 36 continues to receive the relevant parameters of the drone video in flight, the lens PTZ and the aircraft GPS longitude and latitude G2 and altitude h2, calculates the GPS coordinates of the center area of ​​the drone video field of view at that time, and transmits this parameter to the high-position ball camera 24GPS receiving and processing module through the message queue for linkage use in tracking the same target.

[0063] In step one, the operating environment is initialized to prepare the necessary basic data for the normal operation of the system and the implementation of the algorithm, including: obtaining the high-position monitoring video resource table in the visual connection platform 21, which is a directory involving all high-position monitoring, including the device identification of each monitoring channel, the device background login account, login password, network IP, network port, device calibration longitude and latitude, device calibration initial deflection angle, and device viewing angle; obtaining the hangar table in the UAV 23 flight control platform, which includes the hangar IP, device serial number SN code, hangar transmission video stream address, and hangar calibration longitude and latitude; obtaining the event image recognition model library, including the model file of the trained deep neural network image recognition and the path information of the model file stored in the disk; obtaining the high-speed ball camera PTZ initial parameter table, logging in to the ball camera through the program to obtain the value range of the ball camera's horizontal rotation angle P, vertical pitch angle T, and zoom magnification Z, and initialize the ball camera to rotate to the calibrated initial deflection angle.

[0064] In step 2, a deep neural network image recognition algorithm is used to process and analyze the image and video data of the target captured and recorded in real time by the high-position ball camera 24 video, extract the target behavior characteristics, identify the target's motion trajectory and position information, and transmit the information to the PTZ processing submodule 32 through the message queue for processing. The deep neural network image recognition algorithm is based on the YOLO model. The Make Sense tool is used to annotate and classify the collected videos and pictures, and the YOLO model is used to train the annotated pictures and videos, integrate the features of the target, and use geometric transformation and hybrid enhancement technology to optimize the model. The optimized model is evaluated using a test data set, and the optimal model is selected for system integration to achieve image recognition targets captured by the high-position ball camera 24 video.

[0065] In step 3, according to the target detection border information subscribed from the message queue, the PTZ value of the high-position ball camera 24 is automatically adjusted to control the high-position ball camera 24 to track the target, and the target is always placed in the center of the screen. If a single identification target is passed in the read target detection border information, the target detection border center information x, y is taken. If multiple identification targets are passed in, the center information weighted by the confidence rate is taken as the overall center information x, y. At this time, the center information x, y is calculated by the following formula:

[0066]

[0067] After calculating the center information, the program controls the high-position ball camera 24 to move the target center as close to the center of the picture as possible, that is, x=0.5 and y=0.5. In practice, in order to prevent the ball camera from shaking continuously when the target center moves to the center of the picture and cannot lock the target, the aforementioned center of the picture is actually the central area, defined as: x is between 0.4 and 0.6, y is between 0.4 and 0.6. When the target moves to the center area of ​​the picture and lasts for a set time, which is set to 2 seconds here, a locking event is triggered, and the horizontal rotation angle P, vertical pitch angle T, and zoom factor Z of the high-position ball camera 24 are transmitted to the GPS processing submodule 33 through the message queue.

[0068] In step 4, after the GPS processing submodule 33 subscribes to the PTZ value of the high-position ball camera 24 that locks the target from the message queue, it combines the calibrated GPS longitude and latitude of the corresponding high-position ball camera 24 and the relative ground height h1 to determine the current position of the center area of ​​the high-position ball camera 24 lens and the distance a from the high-position ball camera 24, and calculates the GPS coordinates of the target area. The vertical angle A and the horizontal angle B should be the difference between the actual vertical angle A1, the actual horizontal angle B1 and the initial vertical offset angle A2 and the initial horizontal offset angle B2 calibrated by the high-position ball camera 24, which are determined by the following formula:

[0069] A=A1-A2

[0070] B=B1-B2

[0071] Among them, when the high-position ball camera 24 is installed, the horizontal rotation angle is positive from north to east clockwise and negative from counterclockwise; the vertical rotation angle is positive from horizontal downward and negative upward. According to the vertical rotation angle A, according to the equal theorem of alternate angles in parallel lines and the definition of trigonometric functions, the distance a between the projection of the high-position ball camera 24 perpendicular to the ground and the ground target area is obtained by the following formula:

[0072]

[0073] After calculating the distance a, the projection of a in the due east and due north directions is calculated by the trigonometric function of the horizontal deflection angle B, as follows:

[0074] Δx=axsinB

[0075] Δy=axcosB

[0076] Δx and Δy are the offset distances between the longitude and latitude of the detection target area and the calibration coordinates of the high-position ball camera 24. The longitude and latitude of the detection target area are determined by the sum of the longitude and latitude of the calibration coordinates of the high-position ball camera 24 and the offset distance. The conversion relationship between the Δx and Δy offsets and the coordinates is that the longitude Δx differs by 1 meter every 0.00001 degree, and the latitude Δy differs by 1.1 meters every 0.00001 degree. Assuming that the calibration coordinates of the high-position ball camera 24 are G1 and G2, the coordinates G3 and G4 of the detection target area are given by the following formula:

[0077]

[0078] G4=G2+ΔxX0.00001

[0079] The GPS coordinates of the target area are calculated and transmitted to the UAV flight control platform together with the corresponding ball camera parameters and image reasoning event results.

[0080] In step 5, the UAV flight control platform receives the image inference results from each ball camera and the GPS parameters of the target location, generates a UAV flight dispatch route task list based on the event response processing level and the association between GPS and the nearest UAV resources, and issues a task confirmation list to the UAV platform duty personnel. Furthermore, the Euler distance is used to determine the distance, which is determined by the following formula:

[0081]

[0082] Where d is the straight-line distance, x2-x1 is the distance in the X-axis direction, and y2-y1 is the distance in the Y-axis direction.

[0083] In the present invention, a high-position video surveillance resource table, a drone hangar table, an event image recognition model library, and a high-position ball camera 24 PTZ initial parameter table are imported from the visual connection platform 21; the high-position ball camera 24 loads the target event image target detection model through the video target recognition submodule 31, determines the target position in the display area according to the inference result, calculates the PTZ parameters that the ball camera should operate at the next moment according to the preset rules, and passes the parameters to the PTZ processing submodule 32 through the message queue. The above module functions will be executed in a loop until the task is terminated; the PTZ processing submodule 32 cyclically receives the PTZ parameter group sent from other modules, compares the initial PTZ setting of the corresponding high-position ball camera 24, and issues the horizontal, vertical and lens focal length adjustment operation actions of the ball camera, and completes the task. After the action is completed, the PTZ parameters after locking the target are transmitted to the next GPS coordinates; the GPS processing submodule 33 cyclically receives the PTZ parameter group after the target area where the target is located from the PTZ processing submodule 32, collects the GPS longitude and latitude and relative ground height h1 of the corresponding high-position ball camera 24, determines the current position of the central area of ​​the high-position ball camera 24 lens, the distance from the high-position ball camera 24, calculates the GPS coordinates of the target area, and transmits the corresponding ball camera parameters and image reasoning event results to the UAV flight control platform; the UAV flight control platform cyclically receives the image reasoning results from each ball camera and the GPS parameters of the target location G3, and generates according to the event response processing level and the association between GPS and the nearest UAV resources. The UAV flight dispatch route task list is formed, and a task confirmation list is issued to the UAV platform duty personnel; after the personnel confirm that the task UAV 23 takes off according to the route, the MQTT information processing submodule 36 continues to receive the relevant parameters of the UAV video in flight, the lens PTZ and the aircraft GPS longitude and latitude G2 and altitude h2, calculates the GPS coordinates of the center area of ​​the UAV video field of view at that time, and transmits this parameter to the high-position ball camera 24GPS receiving and processing module through the message queue for the same target tracking linkage use. After the duty personnel confirm that the task UAV 23 takes off according to the route, the UAV 23 flight control platform will continue to receive the relevant parameters of the UAV video in flight, the pan-tilt lens PTZ and the aircraft GPS longitude and latitude (G2) and altitude (h2), The same calculation method is used to calculate the GPS coordinates of the target center area of ​​the drone 23 video field of view at that time, and this parameter is transmitted to the high-position ball camera 24 and the GPS receiving and processing module through the message queue for linkage use in the same target tracking. By utilizing the Internet of Things architecture and the transmission message queue technology, the front-end device 2, drone signals, high-position video surveillance streaming media, coordinate calculation, target recognition, video linkage control, and video recognition analysis are integrated into the back-end computing unit, completing the process from discovering and identifying the target through the high-position ball camera 24 video to the linkage drone's continuous visual tracking of the target, realizing the tracking of the drone 23, and the sharing of the perspectives of the drone 23 and the high-position ball camera 24, with complementary advantages; providing a more comprehensive and multi-dimensional solution for emergency monitoring and urban planning.

[0084] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target linkage tracking system based on unmanned aerial vehicle and high-position ball camera video, comprising a target linkage tracking system (1), characterized in that: The target linkage tracking system (1) comprises a front-end device (2), an intermediate processing module (3) and a back-end computing module (4); the front-end device (2) is connected to the intermediate processing module (3), and the intermediate processing module (3) is connected to the back-end computing module (4).

2. The target linkage tracking system based on drone and high-position ball camera video according to claim 1 is characterized in that: The front-end equipment (2) comprises a video link platform (21), a high-position video monitor (22), a drone (23) and a high-position ball camera (24).

3. The target linkage tracking system based on drone and high-position ball camera video according to claim 2 is characterized in that: The intermediate processing module (3) comprises a video target recognition submodule (31), a PTZ processing submodule (32), a GPS processing submodule (33), a drone information receiving submodule (34), a drone task processing submodule (35), an MQTT information processing submodule (36) and a deep neural network submodule (37); the video target recognition submodule (31), the PTZ processing submodule (32) and the GPS processing submodule (33) are all connected to the high-position ball camera (24); and the drone information receiving submodule (34), the drone task processing submodule (35) and the MQTT information processing submodule (36) are all connected to the drone (23).

4. The target linkage tracking system based on drone and high-position ball camera video according to claim 1 is characterized in that: The back-end calculation module (4) comprises a coordinate calculation submodule (41), a target recognition submodule (42), a video linkage control submodule (43) and a video linkage analysis submodule (44).

5. A method for using a target linkage tracking system based on a drone and a high-position ball camera video according to any one of claims 1 to 4, characterized in that: The following steps are involved: Step 1: Initialize the operating environment: import the high-position video surveillance resource table, drone hangar table, event image recognition model library, and high-position ball camera PTZ initial parameter table from the video link platform (21); Step 2, video target recognition: The high-position ball camera (24) loads the target event image target detection model through the video target recognition submodule (31), determines the position of the target in the display area according to the inference result, calculates the PTZ parameters that the ball camera should operate at the next moment according to the pre-set rules, and transmits the parameters to the PTZ processing submodule (32) through the message queue. The above module functions will be executed cyclically until the task is terminated; Step 3, PTZ processing: The PTZ processing submodule (32) cyclically receives the PTZ parameter group sent from other modules, compares it with the initial PTZ setting of the corresponding high-position ball camera (24), issues the horizontal, vertical and lens focal length adjustment operation actions of the ball camera, and after completing the action, transmits the PTZ parameters after locking the target to the first GPS coordinate processing module; Step 4, first GPS processing: The GPS processing submodule (33) cyclically receives the PTZ parameter group after the PTZ processing submodule (32) locks the target area, collects the GPS longitude and latitude and relative ground height h1 of the corresponding high-position ball camera (24), determines the current position of the center area of ​​the high-position ball camera (24), the distance from the high-position ball camera (24), calculates the GPS coordinates of the target area, and transmits the corresponding ball camera parameters and image reasoning event results to the UAV flight control platform; Step 5: Information processing: The UAV flight control platform cyclically receives the image inference results from each ball camera and the GPS parameters of the target location G3, generates a UAV flight dispatch route task list based on the event response processing level and the association between GPS and the nearest UAV resources, and issues a task confirmation list to the UAV platform duty personnel; Step 6, second GPS processing: After the event verification personnel confirm that the mission drone (23) takes off according to the route, the MQTT information processing submodule (36) continues to receive the relevant parameters of the drone video in flight, the lens PTZ and the aircraft GPS longitude and latitude G2 and altitude h2, calculates the GPS coordinates of the center area of ​​the drone video field of view at that time, and transmits this parameter to the high-position ball camera (24) GPS receiving and processing module through the message queue for linkage use in tracking the same target.

6. The method for using the target linkage tracking system based on the video of the drone and the high-position ball camera according to claim 5 is characterized in that: In the step 1, the operating environment is initialized to prepare necessary basic data for the normal operation of the system and the implementation of the algorithm.

7. The method for using the target linkage tracking system based on the video of the drone and the high-position ball camera according to claim 5 is characterized in that: In the step 2, a deep neural network image recognition algorithm is used to process and analyze the image and video data of the target captured and recorded in real time by the high-position ball camera (24), extract the target behavior characteristics, identify the target's motion trajectory and position information, and transmit the information to the PTZ processing submodule (32) through a message queue for processing. The deep neural network image recognition algorithm is based on the YOLO model, uses the Make Sense tool to annotate and classify the collected videos and pictures, uses the YOLO model to train the annotated pictures and videos, integrates the features of the target, uses geometric transformation and hybrid enhancement technology to optimize the model, evaluates the optimized model using a test data set, selects the optimal model for system integration, and realizes the recognition of the image target captured by the high-position ball camera (24) video.

8. The method for using the target linkage tracking system based on the video of the drone and the high-position ball camera according to claim 5 is characterized in that: In the step 3, the PTZ value of the high-position ball camera (24) is automatically adjusted according to the target detection frame information subscribed from the message queue, and the high-position ball camera (24) is controlled to track the target, and the target is always placed in the center of the picture. If a single recognition target is input in the read target detection frame information, the center information (x, y) of the target detection frame is taken. If multiple recognition targets are input, the center information weighted by the confidence rate is taken as the overall center information (x, y). At this time, the center information (x, y) is calculated by the following formula: After the center information is calculated, the program controls the high-position ball camera (24) to move the target center as close to the center of the picture as possible, that is, x=0.5 and y=0.

5. In practice, in order to prevent the ball camera from shaking continuously when the target center moves to the center of the picture and cannot lock the target, the aforementioned picture center is actually the central area, which is defined as: x is between 0.4 and 0.6, and y is between 0.4 and 0.

6. When the target moves to the center area of ​​the picture for a set time, which is set to 2 seconds here (this parameter can be adjusted according to actual needs), the locking event is triggered, and the horizontal rotation angle P, vertical pitch angle T, and zoom magnification Z of the high-position ball camera (24) at this time are transmitted to the GPS processing submodule (33) through the message queue.

9. The method for using the target linkage tracking system based on the video of the drone and the high-position ball camera according to claim 5 is characterized in that: In the fourth step, after the GPS processing submodule (33) subscribes to the PTZ value of the high-position ball camera (24) that locks the target from the message queue, it determines the current position of the center area of ​​the high-position ball camera (24) lens and the distance a from the high-position ball camera (24) in combination with the calibrated GPS longitude and latitude of the corresponding high-position ball camera (24) and the relative ground height h1, and calculates the GPS coordinates of the target area. The vertical angle A and the horizontal angle B should be the difference between the actual vertical angle A1, the actual horizontal angle B1 and the initial vertical offset angle A2 and the initial horizontal offset angle B2 calibrated by the high-position ball camera (24), that is, determined by the following formula: A=A1-A2 B=B1-B2 When the high-position ball camera (24) is installed, the horizontal rotation angle is positive in the clockwise direction from north to east and negative in the counterclockwise direction; the vertical rotation angle is positive in the horizontal downward direction and negative in the upward direction. According to the vertical rotation angle A, according to the equal theorem of alternate angles in parallel lines and the definition of trigonometric functions, the distance a between the projection of the high-position ball camera (24) perpendicular to the ground and the ground target area is obtained by the following formula: After calculating the distance a, the projection of a in the due east and due north directions is calculated by the trigonometric function of the horizontal deflection angle B, as follows: Δx=axsinB Δy=axcosB Δx and Δy are the offset distances of the longitude and latitude between the detection target area and the calibration coordinates of the high-position ball camera (24). The longitude and latitude of the detection target area are determined by the sum of the longitude and latitude of the calibration coordinates of the high-position ball camera (24) and the offset distance. The conversion relationship between the Δx and Δy offsets and the coordinates is that the longitude (Δx) differs by 1 meter for every 0.00001 degree, and the latitude (Δy) differs by 1.1 meters for every 0.00001 degree. Assuming that the calibration coordinates of the high-position ball camera (24) are (G1, G2), the coordinates (G3, G4) of the detection target area are given by the following formula: G4=G2+ΔxX0.00001 The GPS coordinates of the target area are calculated and transmitted to the UAV flight control platform together with the corresponding ball camera parameters and image reasoning event results.

10. The method for using the target linkage tracking system based on the video of the drone and the high-position ball camera according to claim 5, characterized in that: In step 5, the UAV flight control platform receives the image inference results from each ball camera and the GPS parameters of the target location, generates a UAV flight dispatch route task list according to the event response processing level and the association between GPS and the nearest UAV resource, and issues a task confirmation list to the UAV platform on-duty event checker. Further, the Euler distance is used to determine the distance, which is determined by the following formula: Where d is the straight-line distance, x2-x1 is the distance in the X-axis direction, and y2-y1 is the distance in the Y-axis direction.

Images