Aerial unmanned aerial vehicle monitoring system and method based on collaborative awareness and active tracking

Abstract

The invention belongs to the technical field of automatic control, and relates to an air unmanned aerial vehicle monitoring system and method based on collaborative awareness and active tracking, the system comprises an image acquisition module, a calculation control module and a servo control module, the calculation control module comprises a target detection unit, a target tracking unit and an active tracking control unit; the target detection unit comprises a trained improved YOLOv11 model, and is used for inputting a video frame output by the image acquisition module into the trained improved YOLOv11 model to obtain a detection frame in the video frame; the improvement of the improved YOLOv11 model comprises the following steps: replacing a C3k2 module in a backbone network of the YOLOv11 model with a GCM; according to the invention, the C3k2 module in the backbone network of the YOLOv11 model is replaced by the GCM, so that the accuracy and robustness of small target detection are improved.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent security, computer vision and automatic control technology, and relates to an aerial drone monitoring system and method based on collaborative perception and active tracking. Background Technology

[0002] Existing drone monitoring solutions suffer from the following limitations: Disconnect between detection and tracking: Many systems only have detection capabilities and cannot continuously track the drone's movement trajectory, making it difficult to determine its intentions and flight path. Passive monitoring: Using fixed cameras, the system loses the target and monitoring is interrupted when the drone quickly flies out of the monitoring field of view. Poor performance in tracking small targets: General tracking algorithms (such as KCF and MOSSE) are poorly adapted to small-sized drone targets at a distance, easily losing track when the target size changes or is partially obscured. Low system integration: Detection, tracking, and gimbal control are usually treated as independent modules, lacking an efficient collaborative working mechanism, resulting in large system response delays and unstable tracking.

[0003] Therefore, there is an urgent need in this field for an integrated solution that can deeply integrate high-performance detection, robust tracking and active servo control. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides, in one aspect, an aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking, comprising:

[0005] The image acquisition module includes a camera for acquiring video frames of the monitored airspace;

[0006] The calculation and control module is used to generate control commands based on the video frames output by the image acquisition module.

[0007] The servo control module, including the gimbal, is used to drive the gimbal to rotate according to the control commands output by the computing control module; the gimbal is mounted on the camera.

[0008] The calculation control module includes:

[0009] The target detection unit is used to detect UAV targets based on the video frames output by the image acquisition module and output the detection boxes in the video frames.

[0010] The target tracking unit includes the tracked drone trajectory, and is used to track the target of each tracked drone trajectory based on the detection box in the video frame output by the target detection unit, and output the position target information of each tracked drone trajectory;

[0011] The active tracking control unit is used to select a primary tracking target from all tracked drone trajectories and generate control commands based on the location target information of the primary tracking target.

[0012] On the other hand, the present invention employs a collaborative sensing and active tracking method for monitoring aerial unmanned aerial vehicles (UAVs), applied to the aforementioned aerial UAV monitoring system, comprising:

[0013] S1. Acquire real-time video frames of the airspace monitored by the camera;

[0014] S2. Perform UAV target detection based on the acquired video frames to obtain the detection boxes in the video frames;

[0015] S3. Obtain the tracked drone trajectory, perform target tracking on each tracked drone trajectory based on the detection box in the video frame, and output the position target information of each tracked drone trajectory;

[0016] S4. Select a primary tracking target from each tracked drone trajectory according to the preset strategy, and generate control commands based on the location target information of the primary tracking target;

[0017] S5. Drive the gimbal to rotate according to the control command;

[0018] S6. Repeat steps S1 to S5 to achieve continuous and active monitoring of the drone.

[0019] Beneficial effects:

[0020] 1. This invention improves the YOLOv11 model for object detection. The improved YOLOv11 model enhances the feature pyramid structure of the neck network, achieving a more thorough fusion of deep semantic features and shallow detail features through denser connections, thereby improving the accuracy of object detection. 2. This invention replaces the C3k2 module in the backbone network of the YOLOv11 model with GCM. GCM divides the input feature map into two sub-feature maps. and Sub-feature map Inputting the Gaussian sensing module yields a spatial weighted feature map. Sub-feature map The input channel sensing module obtains the channel weight feature map. Spatial weighted feature map AND feature map Multiply, and combine the channel weight feature maps AND feature map Multiplication and weighted combination of the two multiplication results yield the output feature map. Through the cross-interaction mechanism of Gaussian perception and channel attention, background interference is significantly suppressed, allowing feature energy to be concentrated in the real target area. Simultaneously, false responses are eliminated, exhibiting more accurate feature focusing capabilities, thereby improving the accuracy and robustness of small target detection. This provides a stable and reliable input source for the backend DeepSORT algorithm, fundamentally reducing tracking interruptions caused by missed detections. 3. The target tracking unit of this invention uses DeepSORT. DeepSORT's powerful prediction and re-identification capabilities compensate for fluctuations that may occur in single-frame detection, forming a virtuous cycle of "high-quality detection supporting stable tracking, and stable tracking feeding back into continuous monitoring," enabling the system to exhibit extremely high robustness in complex scenarios such as fast movement and brief occlusion. 4. This invention utilizes a high-performance algorithm (i.e., an improved Y...) The combination of the OLOv11 model and DeepSORT with a PTZ pan-tilt unit upgrades a single monitoring device from traditional "fixed field of view" monitoring to "active field of view" monitoring. The pan-tilt unit can drive the camera to perform wide-area scanning. Once a target is detected at the edge of the field of view, it can quickly turn and place it in the center, achieving blind-spot-free monitoring of a large area of ​​airspace. This breaks through the limitations of physical installation location, improves the monitoring efficiency of single-point devices, and reduces the cost of wide-area deployment. 5. This invention is not a simple superposition of algorithms and hardware, but a deep integration through carefully designed software control logic. Data flow between the detection, tracking, and control links is smooth and timely, forming an organic whole. This systematic design brings a "1+1>2" improvement in practical performance, making the entire system more effective, reliable, and practical than any single technology or loosely combined system when dealing with the complex and ever-changing drone intrusion problem in real-world environments. Attached Figure Description

[0021] Figure 1 This is an overall framework diagram of the aerial unmanned aerial vehicle (UAV) monitoring system provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the hardware connection and data flow of the aerial drone monitoring system provided in an embodiment of the present invention;

[0023] Figure 3 An architecture diagram of the improved YOLOv11 model provided for embodiments of the present invention;

[0024] Figure 4 A flowchart illustrating the DeepSORT tracking and active control principle provided in this embodiment of the invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] like Figure 1 , Figure 2 , Figure 4 As shown, this embodiment of the invention employs an aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking, comprising:

[0027] The image acquisition module includes a camera for acquiring video frames of the monitored airspace;

[0028] The image acquisition module uses a high-speed PTZ camera with high resolution (e.g., 4K) and optical zoom capabilities. This device is installed at a high point in the monitored area to ensure a wide field of view. The camera is connected to the computing control module via a network cable.

[0029] The calculation and control module is used to generate control commands based on the video frames output by the image acquisition module.

[0030] The computing control module uses an industrial computer or server equipped with a high-performance GPU (such as NVIDIA Jetson AGX Orin or RTX 4090), which is used to accelerate the inference process of deep learning models. The module receives video streams from the camera via the network port and sends control commands to the gimbal via RS-485 or the network port.

[0031] The calculation control module includes:

[0032] The target detection unit is used to detect UAV targets based on the video frames output by the image acquisition module and output multiple detection boxes in the video frames.

[0033] The object detection unit includes a trained improved YOLOv11 model. By inputting the video frame output by the image acquisition module into the trained improved YOLOv11 model, multiple detection boxes (center point x and y coordinates, width w, height h) and confidence scores in the video frame can be obtained.

[0034] In one embodiment, before inputting the video frames output by the image acquisition module into the trained improved YOLOv11 model, the video frames need to undergo image processing, such as format conversion, cropping, and image enhancement.

[0035] The YOLOv11 model includes: a backbone network, a neck network, and an object detection head; such as... Figure 3As shown, the improvements to the improved YOLOv11 model are as follows: 1. The neck network of the YOLOv11 model is enhanced with upsampling and concatenation connections to achieve multi-scale feature fusion: The feature pyramid structure of the neck network is strengthened, and through denser connections, deeper semantic features and shallower detail features are more fully fused, which is crucial for detecting small drones at a distance; 2. The C3k2 module in the backbone network of the YOLOv11 model is replaced with GCM (Gaussian and Channel Awareness Module): GCM, through the cross-interaction mechanism of Gaussian awareness and channel attention, significantly suppresses background interference, concentrates feature energy in the real target area, and eliminates false responses, exhibiting more accurate feature focusing capabilities, thereby improving the accuracy and robustness of small target detection.

[0036] The core design philosophy of GCM is to enhance the model's representational capabilities through a structured feature decoupling and attention fusion mechanism, including: Gaussian Perception module and Channels Perception module; specifically, the data processing procedure of GCM is as follows:

[0037] Obtain the input feature map X, and then split the input feature map X according to the channel dimension, so that the input feature map X is evenly divided into two sub-feature maps. and ;

[0038] Sub-feature map Inputting the Gaussian sensing module yields a spatial weighted feature map. Sub-feature map The input channel sensing module obtains the channel weight feature map. Both the Gaussian sensing module and the channel sensing module are existing modules.

[0039] The core task of the Gaussian perception module is to analyze sub-feature maps. It performs spatial dimension feature extraction by simulating the properties of a Gaussian distribution and focusing on sub-feature maps. A spatially significant region is generated to create a spatially weighted feature map that characterizes the importance of different pixels. .

[0040] The channel perception module focuses on sub-feature maps The feature evaluation of the channel dimension aims to analyze and weigh the sub-feature maps. The contribution of information carried by each feature channel is used to generate a channel weight feature map that represents the importance of each channel. .

[0041] Spatial weight feature map AND feature map Multiply, and combine the channel weight feature maps AND feature map Multiplication, utilizing learnable scaling parameters and The two product results are weighted and combined to obtain the output feature map. The formula is:

[0042]

[0043] This step allows the network to adaptively adjust the signal strength from spatial and channel paths to achieve information complementarity, thereby improving the robustness of detecting small targets in complex backgrounds.

[0044] Specifically, the backbone network of the improved YOLOv11 model consists of, from top to bottom, a first convolutional layer, a second convolutional layer, a first GCM, a third convolutional layer, a second GCM, a fourth convolutional layer, a third GCM, a fifth convolutional layer, a fourth GCM, SPPF, and C2PSA connected in sequence; where SPPF is a fast spatial pyramid pooling module and C2PSA is a cross-stage partial-pyramid squeeze attention module.

[0045] The improved YOLOv11 model's neck network includes a bottom-up feature fusion module and a top-down feature fusion module.

[0046] The bottom-up feature fusion module includes a first upsampling module, a first stitching module, a first C3k2 module, a second upsampling module, a second stitching module, and a second C3k2 module connected in sequence; the input of the first upsampling module is also connected to the output of C2PSA, the input of the first stitching module is also connected to the output of the third GCM, and the input of the second stitching module is also connected to the output of the second convolutional layer;

[0047] The top-down feature fusion module includes a sixth convolutional layer, a third stitching module, a third C3k2 module, a seventh convolutional layer, a fourth stitching module, and a fourth C3k2 module connected in sequence; the input of the sixth convolutional layer is connected to the output of the second C3k2 module; the top-down feature fusion module also includes a first combining module and a second combining module, the input of the first combining module is connected to the output of the third GCM and the output of C2PSA, and the output of the first combining module is connected to the input of the fourth stitching module; the input of the second combining module is connected to the output of the second GCM and the output of the second upsampling module, and the output of the second combining module is connected to the input of the third stitching module.

[0048] This invention adds a first combination module and a second combination module to the top-down feature fusion module, which enables the neck network and the backbone network to be more densely connected, thereby achieving a more complete fusion of deep semantic features and shallow detail features.

[0049] The improved YOLOv11 model has three target detection heads (Detect), and the input of each target detection head is connected to the output of the second, third, and fourth C3k2 modules, respectively.

[0050] The training of the improved YOLOv11 model includes: collecting and labeling a large-scale aerial drone dataset containing drone images with different weather, lighting, scale, and angle; and using this dataset to train the improved YOLOv11 model.

[0051] The target tracking unit includes the tracked drone trajectory, and is used to track the target of each tracked drone trajectory based on the detection box in the video frame output by the target detection unit, and output the position target information of each tracked drone trajectory;

[0052] The target tracking unit employs the existing DeepSORT (Deep Simple Online and Realtime Tracking) algorithm. The target tracking unit includes:

[0053] The trajectory prediction unit is used to predict the position of each tracked drone trajectory (i.e., drone target) in the current video frame using Kalman filtering;

[0054] For each tracked drone trajectory, Kalman filtering is used to predict its position in the current video frame based on its motion state (position, velocity) in the previous video frame, i.e., the predicted detection box (center point x, y coordinates, width w, height h).

[0055] The association measurement unit is used to perform association measurement between the predicted position of each tracked UAV trajectory output by the trajectory prediction unit in the current video frame and each detection box in the current video frame output by the target detection unit, so as to obtain a comprehensive association cost matrix.

[0056] The correlation measurement units include:

[0057] The Mahalanobis distance unit is used to calculate the Mahalanobis distance between the predicted position of each tracked UAV trajectory j (output by the trajectory prediction unit) in the current video frame and each detection box i (output by the target detection unit) in the current video frame. ;

[0058] Mahalanobis distance measures the spatial consistency between the predicted location and the detection box in the motion model, and has good adaptability to fast linear motion.

[0059] The appearance feature unit is used to crop the corresponding image for each detection box in the current video frame output by the target detection unit. A lightweight ReID network is used to extract features from the image corresponding to each detection box in the current video frame, obtaining the appearance features of each detection box. The historical appearance features of each tracked UAV trajectory j are obtained, and the cosine similarity between the historical appearance features of each tracked UAV trajectory j and the appearance features of each detection box i in the current video frame is calculated. It is used to distinguish different targets with similar appearances and to re-identify targets after they are briefly occluded; among them, the ReID network is the Person Re-identification network.

[0060] The association unit is used to weightedly fuse the Mahalanobis distance output by the Mahalanobis distance unit and the cosine similarity output by the appearance feature unit to obtain the comprehensive association cost matrix. ;in, M and N represent the tracked drone trajectory j and the number of detection boxes i in the current video frame, respectively. As weight.

[0061] The data association unit is used to match each tracked UAV trajectory j with each detection box i in the current video frame using the Hungarian matching algorithm based on the comprehensive association cost matrix, and obtain the matching result; and calculate the position target information (usually coordinates in the image coordinate system) of all tracked UAV trajectories based on the matching result.

[0062] Specifically, the matching results include successfully matched pairs. The process involves identifying unmatched bounding boxes and unmatched trajectories. Based on the matching results, the location target information of all tracked drone trajectories is calculated, including: performing a Kalman update on the successfully matched tracked drone trajectory j to obtain its location target information (usually coordinates in the image coordinate system); adding the appearance features of the bounding boxes matching the tracked drone trajectory j to the feature library of the tracked drone trajectory j; recalculating the historical appearance features (average appearance features) of the tracked drone trajectory j; deleting unmatched drones from the tracked drone trajectories; and creating new drone trajectories based on the unmatched bounding boxes for tracking.

[0063] The active tracking control unit is used to select a primary tracking target from all tracked drone trajectories according to a preset strategy, and generate control commands based on the location target information of the primary tracking target.

[0064] The default strategy is to select the drone trajectory with the highest confidence in the detection box that matches in the current video frame, the largest matching detection box, or the earliest appearance among all tracked drone trajectories as the primary tracking target.

[0065] Generating control commands based on the location information of the primary target includes: calculating the pixel deviation between the center point coordinates of the primary target and the center point of the video frame. A proportional-integral controller is used to convert pixel deviations into angle and speed commands for gimbal rotation. For example: gimbal horizontal rotation speed = ;in, This is the proportionality coefficient. For horizontal pixel deviation, This represents the vertical pixel offset.

[0066] The servo control module, including the gimbal, is used to drive the gimbal to rotate according to the control commands output by the computing control module; the gimbal is mounted on the camera.

[0067] The servo control module receives PTZ commands from the computing control unit and drives the camera to perform precise horizontal and vertical rotation.

[0068] On the other hand, embodiments of the present invention employ a collaborative perception and active tracking method for monitoring aerial unmanned aerial vehicles (UAVs), applied to the aforementioned aerial UAV monitoring system, comprising:

[0069] S1. Acquire real-time video frames of the airspace monitored by the camera;

[0070] S2. Perform UAV target detection based on the acquired video frames to obtain the detection boxes in the video frames;

[0071] S3. Obtain the tracked drone trajectory, perform target tracking on each tracked drone trajectory based on the detection box in the video frame, and output the position target information of each tracked drone trajectory;

[0072] S4. Select a primary tracking target from each tracked drone trajectory according to the preset strategy, and generate control commands based on the location target information of the primary tracking target;

[0073] S5. Drive the gimbal to rotate according to the control command;

[0074] S6. Repeat steps S1 to S5 to achieve continuous and active monitoring of the drone.

[0075] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking, characterized in that, include: The image acquisition module includes a camera for acquiring video frames of the monitored airspace; The calculation and control module is used to generate control commands based on the video frames output by the image acquisition module. The servo control module, including the gimbal, is used to drive the gimbal to rotate according to the control commands output by the computing control module; The gimbal is mounted on the camera; The calculation control module includes: The target detection unit is used to detect UAV targets based on the video frames output by the image acquisition module and output the detection boxes in the video frames. The target tracking unit includes the tracked drone trajectory, and is used to track the target of each tracked drone trajectory based on the detection box in the video frame output by the target detection unit, and output the position target information of each tracked drone trajectory; The active tracking control unit is used to select a primary tracking target from all tracked drone trajectories and generate control commands based on the location target information of the primary tracking target.

2. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 1, characterized in that, The object detection unit includes a trained improved YOLOv11 model. The video frames output by the image acquisition module are input into the trained improved YOLOv11 model to obtain the detection boxes and confidence scores in the video frames. The improvements to the improved YOLOv11 model include replacing the C3k2 module in the backbone network of the YOLOv11 model with GCM. GCM stands for Gaussian and Channel Sensing Module.

3. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 2, characterized in that, GCM includes: a Gaussian sensing module and a channel sensing module; the GCM processing includes: The input feature map is divided into two sub-feature maps. and ; Sub-feature map Inputting the Gaussian sensing module yields a spatial weighted feature map. Sub-feature map The input channel sensing module obtains the channel weight feature map. ; Spatial weight feature map AND feature map Multiply, and combine the channel weight feature maps AND feature map Multiplication, utilizing learnable scaling parameters and The two multiplication results are weighted and combined to obtain the output feature map.

4. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 2, characterized in that, The backbone network of the improved YOLOv11 model consists of, from top to bottom, a first convolutional layer, a second convolutional layer, a first GCM, a third convolutional layer, a second GCM, a fourth convolutional layer, a third GCM, a fifth convolutional layer, a fourth GCM, SPPF, and C2PSA connected in sequence.

5. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 4, characterized in that, The improved YOLOv11 model's neck network includes a bottom-up feature fusion module and a top-down feature fusion module; The bottom-up feature fusion module includes a first upsampling module, a first stitching module, a first C3k2 module, a second upsampling module, a second stitching module, and a second C3k2 module connected in sequence; the input of the first upsampling module is also connected to the output of C2PSA, the input of the first stitching module is also connected to the output of the third GCM, and the input of the second stitching module is also connected to the output of the second convolutional layer; The top-down feature fusion module includes a sixth convolutional layer, a third stitching module, a third C3k2 module, a seventh convolutional layer, a fourth stitching module, and a fourth C3k2 module connected in sequence; the input of the sixth convolutional layer is connected to the output of the second C3k2 module; the top-down feature fusion module also includes a first combining module and a second combining module, the input of the first combining module is connected to the output of the third GCM and the output of C2PSA, and the output of the first combining module is connected to the input of the fourth stitching module; The input of the second combining module is connected to the output of the second GCM and the output of the second upsampling module, and the output of the second combining module is connected to the input of the third splicing module.

6. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 1, characterized in that, The target tracking unit includes: The trajectory prediction unit is used to predict the position of each tracked drone trajectory in the current video frame using Kalman filtering; The association measurement unit is used to perform association measurement between the predicted position of each tracked UAV trajectory output by the trajectory prediction unit in the current video frame and each detection box in the current video frame output by the target detection unit, so as to obtain a comprehensive association cost matrix. The data association unit is used to match each tracked UAV trajectory with each detection box in the current video frame using the Hungarian matching algorithm based on the comprehensive association cost matrix, and to obtain the matching result; and to calculate the location target information of all tracked UAV trajectories based on the matching result.

7. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 6, characterized in that, Selecting a primary tracking target includes: choosing the drone trajectory with the highest confidence in the detection box that matches in the current video frame, the largest matching detection box, or the earliest appearing drone trajectory among all tracked drone trajectories.

8. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 5, characterized in that, The correlation measurement units include: The Mahalanobis distance unit is used to calculate the Mahalanobis distance between the predicted position of each tracked UAV trajectory output by the trajectory prediction unit in the current video frame and the target detection unit output by the target detection unit in the current video frame. The appearance feature unit is used to crop the corresponding image of each detection box in the current video frame output by the target detection unit. A lightweight ReID network is used to extract the features of the image corresponding to each detection box in the current video frame to obtain the appearance features of each detection box. The historical appearance features of each tracked drone trajectory are obtained, and the cosine similarity between the historical appearance features of each tracked drone trajectory and the appearance features of each detection box in the current video frame is calculated. The association unit is used to perform weighted fusion of the Mahalanobis distance output by the Mahalanobis distance unit and the cosine similarity output by the appearance feature unit to obtain the comprehensive association cost matrix.

9. The aerial unmanned aerial vehicle (UAV) monitoring system based on collaborative perception and active tracking according to claim 1, characterized in that, The location information of the primary target being tracked includes: center point coordinates; the control commands generated based on the location information of the primary target being tracked include: calculating the pixel deviation between the center point coordinates of the primary target being tracked and the center point of the video frame, and converting the pixel deviation into angle and speed commands for gimbal rotation.

10. A method for monitoring aerial unmanned aerial vehicles (UAVs) through collaborative perception and active tracking, applied to the aerial UAV monitoring system according to any one of claims 1 to 9, characterized in that, include: S1. Acquire real-time video frames of the airspace monitored by the camera; S2. Perform UAV target detection based on the acquired video frames to obtain the detection boxes in the video frames; S3. Obtain the tracked drone trajectory, perform target tracking on each tracked drone trajectory based on the detection box in the video frame, and output the position target information of each tracked drone trajectory; S4. Select a primary tracking target from each tracked drone trajectory according to the preset strategy, and generate control commands based on the location target information of the primary tracking target; S5. Drive the gimbal to rotate according to the control command; S6. Repeat steps S1 to S5 to achieve continuous and active monitoring of the drone.

Images