Multi-camera multi-unmanned aerial vehicle tracking method based on global attention mechanism

Abstract

The invention provides a multi-camera multi-unmanned aerial vehicle tracking method based on a global attention mechanism, relates to the technical field of video target tracking, and designs a GAM-MTMCT model containing GAM and Ghost convolution. Specifically, a feature extraction network based on ResNet-50 is constructed, a GAM attention mechanism module is introduced, a feature pyramid network FPN is combined with feature maps from different levels, and a region proposal network RPN is used to generate candidate regions on a multi-scale feature map generated by the feature pyramid network FPN; a region proposal network RPN generates candidate frames with different sizes and proportions by sliding a small window on the feature map, and calculates a score and a position regression value for each candidate frame; after the candidate areas are generated, the candidate areas are aligned, and target detection and classification are carried out on the candidate areas subjected to ROI (Region of Interest) Align processing by using Desection Head; and the Desection Head processes the candidate region through Ghost convolution, and outputs the category and position regression information of each candidate box.

Description

Technical Field

[0001] The present invention relates to the field of video target tracking technology, and in particular to a multi-camera multi-UAV tracking method based on a global attention mechanism. Background Art

[0002] With the booming drone industry, a variety of aircraft have gradually become part of everyday life, making them readily available to all walks of life. Drones are widely used in areas such as aerial photography, patrols, surveillance, and light shows. Their compact size and portability make them highly competitive. These significant advantages promise broad future applications and a broad consumer market. After recent years of rapid development and widespread adoption, drone fever has arrived, and the user base continues to expand.

[0003] However, the widespread popularity and use of drones has also brought with it a host of challenges. Drones are relatively easy to operate, requiring no specialized training. Furthermore, their wide adaptability makes them less restricted by terrain and other conditions. Furthermore, as a nascent industry, drone regulations and laws are not fully developed. This makes it difficult to ensure the misuse of drones in private and confidential areas, such as residential and military areas. While some drone manufacturers have established no-fly zones, these restrictions can still be circumvented through software hacking, and not all manufacturers adhere to them. These issues pose a threat to national law enforcement, public safety, and personal privacy.

[0004] Faced with the rapid adoption of drones and the associated problems, society and the government need to take a series of measures to address the challenges. First, to ensure public safety and prevent the misuse of drones, a more stringent and comprehensive legal and regulatory framework is necessary. These laws should include prohibitions on drone operations in private and sensitive areas, and severe penalties for violators. Furthermore, stricter flight permit requirements should be established to ensure that only individuals with professional training and certification can legally operate drones. Second, regulators need to strengthen oversight of the drone market. This includes requiring drone manufacturers to strictly adhere to no-fly zones and ensuring their products are not easily hackable. Regulators can also implement technical means to monitor drone flight paths and remotely control drones in flight to prevent violations. Furthermore, public education is key to addressing drone issues. People need to understand the legal uses and potential risks of drones so they can operate these aircraft more responsibly. Education and awareness campaigns can help raise awareness and reduce the incidence of illegal operations.

[0005] In general, with the rapid development of the drone industry, we must be aware of the potential problems it brings and take measures to address them. To better track drones and achieve drone supervision, this paper proposes a multi-camera multi-drone tracking method based on a global attention mechanism, introducing Ghost convolution and GAM modules. Ghost convolution replaces standard convolution in the ResNet-50 backbone, and the GAM module introduces an attention mechanism after feature extraction to enhance the expressive power of feature maps. Utilizing multi-view video datasets, FPN and RPN network structures effectively fuse multi-view information to achieve more accurate drone tracking. Summary of the Invention

[0006] In view of the problem that existing multi-target tracking algorithms often cannot strike a balance between accuracy and efficiency due to high computational complexity and insufficient feature extraction when handling multi-camera multi-UAV tracking, this paper studies a real-time detection and tracking method for UAVs in a specific area based on multi-camera visual target detection technology, and provides a multi-camera multi-UAV tracking method based on a global attention mechanism.

[0007] A multi-camera multi-drone tracking method based on a global attention mechanism includes the following steps:

[0008] Step 1: Collect and create a multi-view UAV flight video dataset as a training set;

[0009] Specifically, we shoot drone flight videos with multiple cameras, each with the same viewing angle and parameters. We filter and annotate the captured videos to create a multi-view drone flight video dataset.

[0010] Step 2: Build a new tracking model, specifically designing a GAM-MTMCT model that includes GAM and Ghost convolution;

[0011] Step 2.1: Build a feature extraction network based on ResNet-50; Based on the existing ResNet-50 network, replace the standard convolution with Ghost convolution; the first layer Conv1 of the ResNet-50 network uses 7x7 Ghost convolution for preliminary feature extraction; the subsequent layers are Conv2_x, Conv3_x, Conv4_x and Conv5_x layers, each consisting of several Ghost residual blocks, each of which includes two convolution operations and a skip connection; specifically, assuming the input feature map is X0, the output feature map X1 after Ghost convolution is:

[0012] X1=GhostConv(X0)

[0013] Among them, GhostConv represents the Ghost convolution operation;

[0014] Step 2.2: After feature extraction, the GAM attention mechanism module is introduced to improve the network's attention to key features by weighting the features of different channels. The GAM attention mechanism module first processes the input feature map through global average pooling and calculates the global feature vector z:

[0015] z=GlobalAvgPool(X1)

[0016] Then, the global feature vector is used to weight the input feature map to obtain the weighted feature map X′. The weighting formula is as follows:

[0017] X′=X1×σ(Wz)

[0018] Among them, σ represents the Sigmoid activation function, W is the weight matrix, and × represents element-by-element multiplication;

[0019] Step 2.3: Introduce the Feature Pyramid Network (FPN). The Feature Pyramid Network (FPN) enhances the network's perception of objects of different sizes by combining feature maps from different levels. Specifically, the Feature Pyramid Network (FPN) extracts feature maps F from the Conv2_x to Conv5_x layers of the ResNet-50 feature extraction network. i , i∈[2,5], and fuse the low-resolution feature map with the high-resolution feature map by upsampling and lateral connection to generate a multi-scale feature expression and obtain a multi-scale feature map; the fusion operation formula of the feature pyramid network FPN is as follows:

[0020] P i =UpSample(F i+1 )+F i

[0021] Among them, P i is the fused feature map, UpSample represents the upsampling operation;

[0022] Step 2.4: Use the region proposal network RPN to generate candidate regions on the multi-scale feature map generated by the feature pyramid network FPN; the region proposal network RPN generates candidate boxes of different sizes and scales by sliding a small window on the feature map, and calculates its score and position regression value for each candidate box; the score of the target candidate box s is calculated by the sigmoid function, and the regression value Regression is predicted by the regression network; the specific formula is:

[0023] Score(s)=sigmoid(W s ,P i')

[0024] Regression(Δx,Δy,Δw,Δh)=W r ·P i '

[0025] Among them, P i ' is the feature map of the i-th layer, W s and W r are the weights for classification and regression, and (Δx, Δy, Δw, Δh) represents the regression offset of the candidate box position.

[0026] Step 2.5: After generating the candidate region, ROI Align is used to align the candidate region to ensure the alignment accuracy between the candidate frame and the feature map. ROI Align avoids quantization error through bilinear interpolation. Its operation formula is:

[0027]

[0028] Among them, (x1, y1, x2, y2) are the coordinates of the upper left corner and lower right corner of the specified candidate area, and (u, v) represents the sampling point calculated by bilinear interpolation;

[0029] Step 2.6: After the candidate boxes are aligned, the Detection Head is used to perform target detection and classification on the candidate regions processed by ROI Align. The Detection Head processes the candidate regions through Ghost convolution and outputs the category and position regression information of each candidate box, completing the construction of the GAM-MTMCT model.

[0030] The target classification is calculated by the softmax function:

[0031] Classify(X roi )=softmax(W c ·X roi )

[0032] X roi Represents the feature vector of ROI, W c is the weight matrix of the classifier.

[0033] Position regression is done through a regression network:

[0034] Regress(X roi )=W r ·X roi

[0035] Regress means to perform bounding box coordinate regression, W r is a weight matrix for regression.

[0036] Step 3: Train the new tracking model;

[0037] Step 3.1: Randomly extract a batch of multi-view video frames from the training set as input video frames;

[0038] Step 3.2: Perform forward propagation;

[0039] During the forward propagation process, the video frame image is processed by various layers of the new tracking model. The proposed algorithm generates a feature map based on the input data. In the feature extraction module, the ResNet-50 network extracts features from the input video frame image, extracting features at multiple levels including edges, textures, colors, and shapes. These features are then processed and enhanced by the GAM attention mechanism module, the Feature Pyramid Network (FPN), and the Region Proposal Network (RPN) proposal network, ultimately obtaining candidate regions and target feature information in the image.

[0040] Step 3.3: After completing the forward propagation, the new tracking model calculates the loss function based on the current predicted results and the true value;

[0041] The loss function is used to measure the difference between the predicted result and the true value, including classification error, positioning error and regression error. By calculating the value of the loss function, the new tracking model learns the deviation between the current predicted result and the true value, thereby updating the model parameters.

[0042] Step 3.4: Perform back propagation;

[0043] During the back-propagation process, the new tracking model updates its parameters based on the gradient information of the loss function;

[0044] Specifically, backpropagation calculates the gradient of the loss function relative to the weights of each layer through the chain rule, and adjusts the weights in the network through the optimization algorithm, so that the loss function value gradually decreases;

[0045] Step 3.5: Repeat steps 3.2 to 3.4 for multiple iterations until the loss function converges. Each training session uses a different batch of input video frames. After each round of training, the new tracking model updates its model parameters and prepares for the next round of training. When the entire training set is traversed, the training process ends and the new tracking model obtains the optimal model parameters.

[0046] The loss function is as follows:

[0047]

[0048] L total =α·L cls +β·L loc +γ·Ltrack

[0049] Where N is the number of targets, C is the number of categories, J is the number of cameras, and y i,c is the true label, is the predicted probability, p i is the predicted bounding box, t i is the ground-truth bounding box, m ij is the mapping matrix element, which represents the visibility and matching relationship of target i in camera j. α, β, and γ are weight parameters used to adjust the proportion of each loss component in the total loss.

[0050] Step 4: Input the multi-view video to be tracked into the trained new tracking model, which can track multiple drones; finally, it outputs the location information and tracking path of the drones.

[0051] The beneficial effects of adopting the above technical solution are:

[0052] This paper provides a multi-camera, multi-drone tracking method based on a global attention mechanism, introducing Ghost convolution and the GAM module. Ghost convolution replaces standard convolution in the ResNet-50 backbone, reducing computational complexity and improving efficiency. The GAM module introduces an attention mechanism after feature extraction, enhancing the expressiveness of feature maps and improving tracking accuracy and stability. Furthermore, by utilizing a multi-view video dataset and employing FPN and RPN network structures, the method effectively integrates multi-view information to achieve more accurate drone tracking.

[0053] Through the above technical solutions, the multi-camera, multi-drone tracking method provided by the present invention, based on GAM and Ghost convolution, can accurately track multiple drones in complex environments. The use of Ghost convolution and GAM modules improves the model's computational efficiency and feature expression capabilities, enabling the system to effectively track drones under various environmental conditions. By fully utilizing multi-view information, the tracking algorithm of the present invention can maintain high accuracy and robustness despite occlusion and complex backgrounds, providing strong technical support for the safe flight and monitoring of drones.

[0054] The present invention effectively solves the problems of high computational complexity and insufficient feature extraction in the prior art by introducing Ghost convolution and GAM modules. Through training with multi-view video datasets, the model of the present invention can achieve accurate tracking of multiple drones in complex environments, especially in real-time monitoring and management of key areas, with significant practical application value and technical advantages. The purpose of the present invention is to improve the real-time detection and tracking capabilities of drones in specific environments through multi-camera visual target detection technology, thereby optimizing the monitoring coverage, ensuring that drone activities in sensitive areas are effectively monitored, and providing technical guarantees for public safety and privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A novel tracking model structure framework provided by an embodiment of the present invention;

[0056] Figure 2 This is a structural diagram of the GAM attention mechanism module in an embodiment of the present invention;

[0057] Figure 3 2 is a structural diagram of the FPN module in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0059] A multi-camera multi-drone tracking method based on a global attention mechanism includes the following steps:

[0060] Step 1: Collect and create a multi-view UAV flight video dataset as a training set;

[0061] Specifically, we shoot drone flight videos with multiple cameras, each with the same viewing angle and parameters. We filter and annotate the captured videos to create a multi-view drone flight video dataset.

[0062] Step 2: Build a new tracking model, specifically designing a GAM-MTMCT model that includes GAM and Ghost convolution;

[0063] Step 2.1: Construct a feature extraction network based on ResNet-50; based on the existing ResNet-50 network, replace the standard convolution with Ghost convolution; the Ghost convolution reduces the amount of computation by generating a Ghost feature map, maintaining a high feature extraction capability. The first layer Conv1 of the ResNet-50 network uses a 7x7 Ghost convolution for preliminary feature extraction; the subsequent layers are Conv2_x, Conv3_x, Conv4_x, and Conv5_x layers, each consisting of several Ghost residual blocks, each of which includes two convolution operations and a jump connection; the design of this structure reduces the amount of computation through Ghost convolution, while improving the efficiency and feature extraction capability of the network. Specifically, assuming that the input feature map is X0, the output feature map X1 after Ghost convolution is:

[0064] X1=GhostConv(X0)

[0065] Among them, GhostConv represents the Ghost convolution operation. This design provides the basic feature representation for subsequent modules and optimizes the computing performance. Figure 1 As shown in Figure 2, the structural framework of the new tracking model is shown.

[0066] Step 2.2: After feature extraction, the GAM (Global Attention Module) attention mechanism module is introduced to improve the network's attention to key features by weighting the features of different channels. The GAM attention mechanism module first processes the input feature map through global average pooling (GlobalAvgPool) to calculate the global feature vector z:

[0067] z=GlobalAvgPool(X1)

[0068] Then, the global feature vector is used to weight the input feature map to obtain the weighted feature map X′. The weighting formula is as follows:

[0069] X′=X1×σ(Wz)

[0070] Among them, σ represents the Sigmoid activation function, W is the weight matrix, and × represents element-by-element multiplication;

[0071] The GAM module can weight the channels, highlight the features of important channels, suppress unimportant or redundant information, and enhance the expressiveness of feature maps. Figure 2 The structure of the GAM module is presented, illustrating how key information is enhanced through the attention mechanism.

[0072] Step 2.3: In order to improve the detection capability of multi-scale targets, the feature pyramid network FPN is introduced. The feature pyramid network FPN enhances the network's perception of targets of different sizes by combining feature maps from different levels. Specifically, the feature pyramid network FPN extracts feature maps F from Conv2_x to Conv5_x layers of the feature extraction network of ResNet-50. i , i∈[2,5], and fuse the low-resolution feature map with the high-resolution feature map by upsampling and lateral connection to generate a multi-scale feature expression and obtain a multi-scale feature map; the fusion operation formula of the feature pyramid network FPN is as follows:

[0073] P i =UpSample(F i+1 )+F i

[0074] Among them, P i is the fused feature map, UpSample represents the upsampling operation;

[0075] Through this multi-scale fusion, FPN enhances the detection capability of objects of different sizes and improves the accuracy of object detection. Figure 3 The structure of the FPN module is presented, showing how to enhance object detection capabilities through multi-scale feature fusion.

[0076] Step 2.4: Use the region proposal network RPN to generate candidate regions on the multi-scale feature map generated by the feature pyramid network FPN; the region proposal network RPN generates candidate boxes of different sizes and scales by sliding a small window on the feature map, and calculates its score and position regression value for each candidate box; the score of the target candidate box s is calculated by the sigmoid function, and the regression value Regression is predicted by the regression network; the specific formula is:

[0077] Score(s)=sigmoid(W s ,P i ')

[0078] Regression(Δx,Δy,Δw,Δh)=W r ·P i '

[0079] Among them, P i ' is the feature map of the i-th layer, W s and W r are the weights for classification and regression, and (Δx, Δy, Δw, Δh) represents the regression offset of the candidate box position.

[0080] RPN generates candidate boxes that provide potential regions for subsequent target detection. This process provides preliminary candidate regions for target detection tasks.

[0081] Step 2.5: After generating the candidate region, use the ROI Align method to align the candidate region to ensure the alignment accuracy between the candidate frame and the feature map; the traditional ROI pooling method will cause quantization error, affecting the detection accuracy. ROI Align avoids quantization error through bilinear interpolation and ensures the accurate alignment of the candidate region. Its operation formula is:

[0082]

[0083] Among them, (x1, y1, x2, y2) are the coordinates of the upper left corner and lower right corner of the specified candidate area, and (u, v) represents the sampling point calculated by bilinear interpolation;

[0084] Through this method, the feature maps of all candidate boxes have the same size, ensuring the accuracy of each candidate box during the target detection process.

[0085] Step 2.6: After the candidate boxes are aligned, the Detection Head is used to perform target detection and classification on the candidate regions processed by ROI Align. The Detection Head processes the candidate regions through Ghost convolution and outputs the category and position regression information of each candidate box, completing the construction of the GAM-MTMCT model.

[0086] The target classification is calculated by the softmax function:

[0087] Classify(X roi )=softmax(W c ·X roi )

[0088] X roi Represents the feature vector of ROI, W c is the weight matrix of the classifier.

[0089] Position regression is done through a regression network:

[0090] Regress(X roi )=W r ·X roi

[0091] Regress means to perform bounding box coordinate regression, W r is a weight matrix for regression.

[0092] These operations provide category prediction and position regression values ​​for each candidate region, completing the detection and positioning of the target.

[0093] Step 3: Train the new tracking model;

[0094] Step 3.1: When training the new tracking model, first randomly select a batch of multi-view video frames from the training set as input. The video frames selected should be diverse and representative to ensure that the model can handle object tracking tasks from different viewpoints. These video frames are passed as input to the new tracking model for training.

[0095] Step 3.2: Perform forward propagation;

[0096] During the forward propagation process, the video frame image is processed by various layers of the new tracking model. The proposed algorithm generates a feature map based on the input data. In the feature extraction module, the ResNet-50 network extracts features from the input video frame image, extracting features at multiple levels including edges, textures, colors, and shapes. These features are then processed and enhanced by the GAM attention mechanism module, the Feature Pyramid Network (FPN), and the Region Proposal Network (RPN) proposal network, ultimately obtaining candidate regions and target feature information in the image.

[0097] Step 3.3: After completing the forward propagation, the new tracking model calculates the loss function based on the current predicted results and the true value;

[0098] The loss function is used to measure the difference between the predicted result and the true value, including classification error, positioning error and regression error. By calculating the value of the loss function, the new tracking model learns the deviation between the current predicted result and the true value, thereby updating the model parameters.

[0099] Step 3.4: Perform back propagation;

[0100] During the back-propagation process, the new tracking model updates its parameters based on the gradient information of the loss function;

[0101] Specifically, backpropagation uses the chain rule to calculate the gradient of the loss function with respect to the weights of each layer. It then uses an optimization algorithm (such as SGD or Adam) to adjust the weights in the network, gradually reducing the loss function and improving model performance. Specifically, the network gradually adjusts the weights at each layer, so that after each training session, the model's output is closer to the true target label.

[0102] Step 3.5: Repeat steps 3.2 to 3.4 for multiple iterations until the loss function converges. Each training session uses a different batch of input video frames. After each round of training, the new tracking model updates its model parameters and prepares for the next round of training. When the entire training set is traversed, the training process ends and the new tracking model obtains the optimal model parameters, which enables better target tracking capabilities.

[0103] The loss function is as follows:

[0104]

[0105] L total =α·L cls +β·L loc +γ·L track

[0106] Where N is the number of targets, C is the number of categories, J is the number of cameras, and y i,c is the true label, is the predicted probability, p i is the predicted bounding box, t i is the ground-truth bounding box, m ij is the mapping matrix element, which represents the visibility and matching relationship of target i in camera j. α, β, and γ are weight parameters used to adjust the proportion of each loss component in the total loss.

[0107] Step 4: Input the multi-view video to be tracked into the trained new tracking model, which can track multiple drones; finally, it outputs the location information and tracking path of the drones.

[0108] The GAM global attention mechanism module introduced in this embodiment can amplify the global dimension interaction features while reducing information diffusion, which helps to capture important feature information, enhance the expressiveness of feature maps, and improve the performance of the model. The specific process is as follows Figure 2 As shown, the formula is as follows:

[0109] Given an input feature map F1∈R C*H*W , the intermediate state F2 and output F3 are defined as:

[0110]

[0111]

[0112] Among them, C is the number of channels, H and W are the height and width of the feature map respectively, and M C and M S Represent channel attention and spatial attention mechanisms respectively, Represents the element-wise product operation. Through this operation, the GAM module can establish connections between different feature levels and improve the expressiveness and discriminative power of the feature map.

[0113] The introduction of Ghost convolution reduces computational complexity and memory consumption while maintaining efficient feature extraction. Specifically, Ghost convolution generates a sparse feature map and multiplies it element-by-element to generate the final feature map; its calculation formula is as follows:

[0114]

[0115] Among them, N represents the number of sparse feature maps generated in Ghost convolution, F g Represents the final generated Ghost feature map, G i Represents the generated sparse feature map, W i Represents the corresponding weight matrix. In this way, Ghost convolution can effectively reduce the amount of calculation while retaining important feature information and improving the operating efficiency of the model.

[0116] Based on the feature extraction of this system, a Feature Pyramid Network (FPN) is used to further process the feature maps, combining feature maps from different levels to generate more representative features. The FPN design enables the model to fuse features at different scales, capturing more contextual information and thus improving tracking accuracy.

[0117] Specifically, FPN gradually integrates high-level features into low-level features through a top-down path and side connections. This ensures that features at each level contain rich semantic and detailed information. For example, high-level features typically have strong semantic information and less spatial detail, while low-level features contain rich spatial detail and less semantic information. Through the design of FPN, the system is able to integrate these features at different scales, ensuring that feature maps at each level contain rich semantic and detailed information. This design greatly enhances the model's ability to track targets, especially when dealing with multi-scale targets.

[0118] After the FPN network generates a multi-scale feature map, the Region Proposal Network (RPN) is used to generate candidate regions, and ROI Align is used to align the candidate regions to a uniform size for subsequent processing. The RPN network generates a large number of candidate regions on the feature map through a sliding window mechanism, and scores and regresses each candidate region to filter out the region most likely to contain the target from the candidate region. Specifically, the RPN network slides a window on the feature map through a set of anchor points, generating candidate regions of multiple scales and aspect ratios for each position where the sliding window passes. These candidate regions are processed by two fully connected layers, one for classification (determining whether the region contains the target) and the other for regression (adjusting the precise boundaries of the candidate region).

[0119] To ensure consistency and accuracy in subsequent processing, the ROI Align operation is used to align candidate regions of different sizes to the same size. ROI Align uses bilinear interpolation to calculate precise feature values ​​at each location in the candidate region, rather than simply performing quantization. This maximizes the preservation of spatial information, thereby improving the accuracy of target detection and tracking.

[0120] In the Detection Head, Ghost convolution is used for object detection and classification. The Detection Head is responsible for detailed analysis of the aligned candidate regions, including object category identification and bounding box localization. Ghost convolution is a lightweight convolution operation that reduces computational effort and parameter requirements by generating redundant features, thereby improving computational efficiency. Specifically, Ghost convolution first uses standard convolution to generate a small number of primary features, and then generates additional features through linear transformations, significantly reducing computational effort while maintaining feature expressiveness.

[0121] During the model training phase, a batch of multi-view drone image video frames is randomly sampled from the training set and fed into the improved GAM-MTMCT model. First, forward propagation is performed to calculate the loss function and update the model parameters to measure model performance. The loss function typically consists of a classification loss and a regression loss. The classification loss is used to assess the accuracy of target class predictions, while the regression loss is used to assess the accuracy of bounding box predictions. Backward propagation is then performed to calculate the gradient of the loss function and update the model parameters. The model repeats forward and backward propagation until the loss function converges. The entire training process concludes after the entire training set has been traversed, ensuring that the model has fully learned the information contained in the multi-view drone data.

[0122] To improve training efficiency and effectiveness, data augmentation techniques, such as random cropping, rotation, flipping, and color jittering, are used to further enrich the drone training dataset and enhance the generalization capabilities of the GAM-MTMCT model. By generating diverse training samples, data augmentation enables the model to better adapt to the effects of varying environments and noise variations on drone target monitoring, improving predictive performance for unknown data.

[0123] During the model application phase, the drone images to be detected are fed into the trained GAM-MTMCT model for detection and tracking. Specifically, the input multi-view drone images are preprocessed, including image scaling and normalization. Scaling ensures that the input image size meets the model's requirements, while normalization adjusts the pixel value range so that features from different images are compared at the same scale, thereby improving the model's robustness.

[0124] The preprocessed image is then fed into a feature extraction network to extract multi-level image features. This step utilizes a deep convolutional neural network (CNN) to process the image layer by layer, gradually abstracting features from low-level to high-level, extracting a feature map containing rich semantic information. The extracted feature map is then fed into the GAM module to further enhance its expressive power. The GAM module calculates the global relationships between each location in the feature map and generates a weighted matrix that weights the feature map, thereby highlighting important features and suppressing irrelevant information.

[0125] The FPN network generates multi-scale feature maps. Based on multi-layer feature maps, the FPN network gradually integrates high-level features into lower-level features through top-down paths and side connections, ensuring that feature maps at each scale contain rich semantic information and spatial details. This ensures that the model can accurately detect and identify targets at different scales. This is particularly true for drone targets of varying sizes and shapes, as the FPN's multi-scale feature maps provide more precise descriptions.

[0126] Subsequently, the RPN network is used to generate candidate regions, which are then aligned using ROIAlign. The RPN network generates a large number of candidate regions by sliding a window across the feature map. These regions are then classified and regressed to select the candidate regions most likely to contain the object. The ROIAlign operation uses precise bilinear interpolation to align these candidate regions to a uniform size, ensuring consistency and accuracy in subsequent detection and classification steps.

[0127] Finally, in the Detection Head, Ghost convolution is used to further process the candidate regions to generate the final detection results, including the target category and bounding box location. Ghost convolution reduces the amount of computation and parameters by generating redundant features, thereby improving computational efficiency. Specifically, Ghost convolution first uses standard convolution to generate a smaller number of primary features, and then generates additional features through linear transformation, significantly reducing the amount of computation while maintaining the expressiveness of the features. This design enables the Detection Head to accurately identify and locate targets while performing efficient computations, thereby improving the detection performance of the entire system.

[0128] In the multi-camera data integration stage, the system integrates the information of each camera and integrates the tracking information between different cameras with the help of the global attention mechanism (GAM). The global attention mechanism of GAM can amplify the global dimensional interaction features while reducing information diffusion. Specifically, GAM helps the system obtain global information on the entire data set and performs weighted processing based on the importance of different parts, thereby improving the system's understanding of the global context. For the multi-target tracking (MTMCT) process, the trajectory-based dynamic cross-modal correlation algorithm (GAM-MTMCT) is used. The specific formula is as follows:

[0129] 1. Trajectory to target assignment:

[0130]

[0131] 2. Cross-modal association:

[0132]

[0133] Among them, D ij represents the allocation cost between trajectory i and target j, f t i represents the feature representation of trajectory i at time t, g t j represents the feature representation of target j at time t, T represents the total time step (or number of frames), ||·|| represents the distance metric between features, and S ij represents the cross-modal association score between track i and target j;

[0134] By introducing GAM, the system can more accurately identify and associate targets, enabling continuous tracking across different cameras. This integration and weighted processing provides the system with more complete and comprehensive target recognition and tracking capabilities, effectively improving the performance of drone tracking and surveillance systems in multi-camera environments.

[0135] The system continuously iterates, continuously tracking drones and comprehensively monitoring their movements in multi-camera scenarios. Through continuous tracking and monitoring, the system accurately captures the drone's location and trajectory, continuously adjusting and optimizing its predictions throughout the process to ensure continued focus and tracking of the target.

[0136] In summary, this system utilizes a variety of advanced technologies, including FPN, RPN, ROI Align, Ghost convolution, and GAM, to form a complete and efficient multi-camera drone tracking and surveillance system. The FPN network's multi-scale feature fusion capability enables the system to accurately detect and identify targets at different image scales; the RPN network generates high-quality candidate regions and ensures consistent and accurate processing through ROI Align; the efficient computational characteristics of Ghost convolution improve the system's detection performance; and the GAM module, through a global attention mechanism, enhances the system's understanding and processing capabilities of global context. The organic combination of these technologies enables the system to achieve high-precision, continuous drone target tracking and surveillance in a multi-camera environment, significantly improving the system's overall performance.

[0137] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multi-camera multi-UAV tracking method based on global attention mechanism, characterized in that: The following steps are involved: Step 1: Collect and create a multi-view UAV flight video dataset as a training set; Step 2: Build a new tracking model, specifically designing a GAM-MTMCT model that includes GAM and Ghost convolution; Step 2.1: Build a feature extraction network based on ResNet-50. Based on the existing ResNet-50 network, replace the standard convolution with Ghost convolution. Step 2.2: After feature extraction, introduce the GAM attention mechanism module to improve the network's attention to key features by weighting the features of different channels; The GAM attention mechanism module first processes the input feature map X1 through global average pooling and calculates the global feature vector z: z=GlobalAvgPool(X1) Then, the global feature vector is used to weight the input feature map to obtain the weighted feature map X′. The weighting formula is as follows: X′=X1×σ(Wz) Among them, σ represents the Sigmoid activation function, W is the weight matrix, and × represents element-by-element multiplication; Step 2.3: Introduce the Feature Pyramid Network (FPN). The Feature Pyramid Network (FPN) enhances the network’s ability to perceive objects of different sizes by combining feature maps from different levels. Step 2.4: Use the region proposal network RPN to generate candidate regions on the multi-scale feature map generated by the feature pyramid network FPN; the region proposal network RPN generates candidate boxes of different sizes and scales by sliding a small window on the feature map, and calculates its score and position regression value for each candidate box; the score of the target candidate box s is calculated by the sigmoid function, and the regression value Regression is predicted by the regression network; the specific formula is: Score(s)=sigmoid(W s ,P i ′) Regression(Δx,Δy,Δw,Δh)=W r ·P i ' Among them, P i ' is the feature map of the i-th layer, W s and W r are the weights for classification and regression, (Δx, Δy, Δw, Δh) represents the regression offset of the candidate box position; Step 2.5: After generating the candidate region, ROI Align is used to align the candidate region to ensure the alignment accuracy between the candidate frame and the feature map. ROI Align avoids quantization error through bilinear interpolation. Its operation formula is: Among them, (x1, y1, x2, y2) are the coordinates of the upper left corner and lower right corner of the specified candidate area, and (u, v) represents the sampling point calculated by bilinear interpolation; Step 2.6: After the candidate boxes are aligned, the Detection Head is used to perform target detection and classification on the candidate regions processed by ROI Align. The Detection Head processes the candidate regions through Ghost convolution and outputs the category and position regression information of each candidate box, completing the construction of the GAM-MTMCT model. Step 3: Train the new tracking model; Step 3.1: Randomly extract a batch of multi-view video frames from the training set as input video frames; Step 3.2: Perform forward propagation; During the forward propagation process, the video frame image is processed by various layers of the new tracking model. The proposed algorithm generates a feature map based on the input data. In the feature extraction module, the ResNet-50 network extracts features from the input video frame image, extracting features at multiple levels including edges, textures, colors, and shapes. These features are then processed and enhanced by the GAM attention mechanism module, the feature pyramid network (FPN), and the region proposal network (RPN) proposal network, ultimately obtaining candidate regions and target feature information in the image. Step 3.3: After completing the forward propagation, the new tracking model calculates the loss function based on the current predicted results and the true value; Step 3.4: Perform backpropagation. During the backpropagation process, the new tracking model updates its parameters based on the gradient information of the loss function. Step 3.5: Repeat steps 3.2 to 3.4 for multiple iterations until the loss function converges. Each training session uses a different batch of input video frames. After each round of training, the new tracking model updates its model parameters and prepares for the next round of training. When the entire training set is traversed, the training process ends and the new tracking model obtains the optimal model parameters. Step 4: Input the multi-view video to be tracked into the trained new tracking model, which can track multiple drones; finally, it outputs the location information and tracking path of the drones.

2. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: Specifically, step 1 includes shooting the flight video of the drone using multiple cameras, where the viewing angle and parameters of each camera are consistent; screening and labeling the shot videos to create a multi-view drone flight video dataset.

3. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: Specifically, step 2.1 is as follows: the first layer Conv1 of the ResNet-50 network uses 7x7 Ghost convolution for preliminary feature extraction; the subsequent layers are Conv2_x, Conv3_x, Conv4_x and Conv5_x layers, each of which is composed of several Ghost residual blocks, each of which includes two convolution operations and a skip connection; specifically, assuming that the input feature map is X0, the output feature map X1 after Ghost convolution is: X1=GhostConv(X0) Among them, GhostConv represents the Ghost convolution operation.

4. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: Specifically, the feature pyramid network FPN extracts the feature map F from the Conv2_x to Conv5_x layers of the feature extraction network of ResNet-50. i , i∈[2,5], and fuse the low-resolution feature map with the high-resolution feature map by upsampling and lateral connection to generate a multi-scale feature expression and obtain a multi-scale feature map; the fusion operation formula of the feature pyramid network FPN is as follows: P i =UpSample(F i+1 )+F i Among them, P i It is the fused feature map, and UpSample represents the upsampling operation.

5. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: Specifically, step 2.6 is to calculate the target classification through the softmax function: Classify(X roi )=softmax(W c ·X roi ) X roi Represents the feature vector of ROI, W c is the weight matrix of the classifier; Position regression is done through a regression network: Regress(X roi )=W r ·X roi Regress means to perform bounding box coordinate regression, W r is a weight matrix for regression.

6. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: The loss function described in step 3.3 is used to measure the difference between the predicted result and the true value, including classification error, positioning error and regression error; by calculating the value of the loss function, the new tracking model knows the deviation between the current predicted result and the true value, thereby updating the model parameters.

7. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: Step 3.4 is as follows: Back propagation calculates the gradient of the loss function relative to the weights of each layer through the chain rule, and adjusts the weights in the network through the optimization algorithm so that the loss function value gradually decreases.

8. The multi-camera multi-UAV tracking method based on global attention mechanism according to claim 1 is characterized in that: The loss function described in step 3.5 is as follows: Where N is the number of targets, C is the number of categories, J is the number of cameras, and y i,c is the true label, is the predicted probability, p i is the predicted bounding box, t i is the ground-truth bounding box, m ij is the mapping matrix element, which represents the visibility and matching relationship of target i in camera j. α, β, and γ are weight parameters used to adjust the proportion of each loss component in the total loss.

Images