A multi-modal classification-based hybrid tracking method for a target photographed by a UAV

Abstract

The application discloses a hybrid tracking method for a UAV shooting target based on multi-modal classification and relates to the technical field of UAV aerial photography, in particular to a hybrid tracking method for a UAV shooting target based on multi-modal classification. The application adopts a double-tower deep learning architecture, adopts a point cloud branch+vision branch to perform multi-modal feature fusion classification, and selects a SiamFC++ improved module and a TR-TBP-TPMB model for tracking according to a classification result, so that point target tracking results and extended target tracking results are finally obtained. The method comprises the following steps: inputting preprocessed image data sets into a PointNet++ and a ResNet double-tower classification structure, the double-tower classification structure classifies targets into point targets and extended targets; inputting the point targets into a SiamFC++ improved module to obtain point target tracking results; and inputting the extended targets into a TR-TBP-TPMB model to obtain extended target tracking results.

Description

Technical Field

[0001] This invention relates to the field of drone aerial photography technology, and specifically to a hybrid tracking method for drone-captured targets based on multimodal classification. Background Technology

[0002] Target tracking algorithms for UAV aerial photography scenarios have promising applications. UAV aerial targets typically exhibit characteristics such as small size, significant scale variations, and frequent viewpoint changes, making UAV target tracking tasks quite challenging. Furthermore, existing tracking algorithms do not fully utilize inter-frame information about the target, such as its trajectory, speed, and appearance changes. Therefore, designing high-performance UAV target tracking algorithms is of great significance and value.

[0003] The Chinese invention patent "A Multi-Target Tracking Method Based on BP-PMBM Filtering Algorithm" (CN202010185729.6) extends the Poisson-Do-Bernoulli Mixture (PMBM) filtering method to Tag Random Finite Set (LRFS) for track tracking, and adopts the box particle (BP) filtering method to form the BP-PMBM filtering algorithm. This solves the track differentiation problem and improves tracking accuracy and computation speed. However, this patent still has the following shortcomings:

[0004] (1) The problem of the interval expansion of the box particle (in nonlinear systems or long-term tracking, the interval of the box particle will expand continuously over time (due to noise accumulation and state transition), which leads to an increase in the fuzziness of the state estimation and a decrease in accuracy;

[0005] (2) The patent has limitations in adaptability to complex scenarios, and its performance optimization is insufficient in dense target and high clutter environments. It does not cover extreme situations and boundary conditions in practical applications.

[0006] (3) Limitations of the “track differentiation” capability: This patented tag relies solely on measurement index and time. In scenarios such as target intersection and occlusion, tag confusion is likely to occur, leading to track breakage or incorrect association.

[0007] To address the shortcomings of the aforementioned patents, the paper "Trajectory PMB Filters for Extended Object Tracking Using Belief Propagation" (arXiv:2207.10164v3[eess.SP] September 19, 2023, Yuxuan Xia et al.) proposes the following improvement method:

[0008] (1) Particle representation is used instead of box particles to improve the accuracy of state estimation;

[0009] (2) For target-dense scenarios, messages are passed in the factor graph through backpropagation; for extended targets, the problem of multi-measurement generation is handled by combining the Poisson space model.

[0010] (3) The “Track PMB (TPMB) Filter” is proposed. Through smoothing techniques such as backward simulation, the historical trajectory error can be corrected and the track breakage caused by label confusion can be avoided.

[0011] While the aforementioned paper addresses the shortcomings of the aforementioned patent, two problems still exist:

[0012] (1) When tracking multiple targets, if point targets and extended targets coexist, the PMB filter assumes that each target generates at most one measurement point due to the measurement generation mechanism. It will misinterpret the multiple measurement points generated by the extended targets as multiple independent point targets, resulting in errors such as overestimation of the number of targets and decreased tracking accuracy.

[0013] (2) The “near constant speed model” used in the BP-TMPB model in this paper assumes that the target’s speed changes little (approximately constant). However, when the target undergoes complex maneuvers such as rapid acceleration, deceleration, or sharp turns, the model cannot accurately predict the drastic changes in the target’s motion state, which can lead to problems such as increased tracking error and easy loss of the target. Therefore, it is not adaptable to complex maneuvers. Summary of the Invention

[0014] To address the aforementioned problems, this invention proposes a hybrid tracking method for drone-captured targets based on multimodal classification. A dual-tower deep learning architecture is employed, using a point cloud branch and a visual branch for multimodal feature fusion and classification. Based on the classification results, an improved SiamFC++ module and a TR-TBP-TPMB model are selected for tracking, ultimately yielding point target tracking results and extended target tracking results.

[0015] The method includes the following steps:

[0016] S1. Collect motion image datasets of the target using a drone equipped with a lidar, and preprocess the datasets.

[0017] S2. Input the preprocessed dataset into the PointNet++ and ResNet dual-tower classification structure. The dual-tower classification structure classifies the target into point targets and extended targets.

[0018] The PointNet++ and ResNet dual-tower classification architecture includes a PointNet++ point cloud branch, a ResNet vision branch, and an attention layer;

[0019] The PointNet++ point cloud branch passes through the sampling layer, feature aggregation layer, global feature layer, and density feature calculation layer sequentially from input to output.

[0020] The ResNet visual branch passes through the residual layer, the global average pooling layer, the first fully connected layer, and the second fully connected layer in sequence from input to output.

[0021] The attention layer passes through the following layers from input to output: the first fully connected layer, the first weight calculation module, the second fully connected layer, and the Softmax activation function.

[0022] S3. Input the point target into the SiamFC++ improved module to obtain the point target tracking result;

[0023] S4. Extend the target input to the TR-TBP-TPMB model to obtain the extended target tracking results.

[0024] Furthermore, the TR-TBP-TPMB model is obtained by improving the BP-PMBM model, and the improvement is specifically as follows:

[0025] S41. In the prediction branch of the BP-PMBM model, replace the near-constant velocity module with the Transformer module.

[0026] S42. In the update branch of the BP-PMBM model, replace the BP module with the time-series BP module.

[0027] Furthermore, the SiamFC++ improvement module specifically includes:

[0028] In the feature extraction part of the SiamFC++ module, ResNet-18 is truncated to the conv3 layer, and an SE channel attention module is added after the conv3 layer;

[0029] Expand the size of the SiamFC++ module template area: Set the size of the SiamFC++ module template area to twice the size of the target box;

[0030] In the target matching process of the SiamFC++ module, spatial constraints are imposed on the matching score;

[0031] A multi-resolution feature fusion mechanism and a multi-scale adaptive matching strategy are introduced into the search branch of the SiamFC++ module.

[0032] Furthermore, the spatial constraint applied to the matching score during the target matching process in the SiamFC++ module is specifically as follows: when the Euclidean distance between the candidate position and the historical target center position exceeds a preset threshold, the matching score of the candidate position is attenuated, and the value of the preset threshold is twice the target diameter.

[0033] Furthermore, the multi-resolution feature fusion mechanism specifically involves upsampling low-resolution features to restore them to a high-resolution scale, and then stitching them together with other high-resolution features.

[0034] Furthermore, the multi-scale adaptive matching strategy is as follows: in the current frame, several scale versions of the template features are generated, and each scale template is matched with the fused features of the corresponding scale in the search area. Finally, the scale version with the highest matching score is selected from the several scale matching results as the target localization result of the current frame.

[0035] Furthermore, the target's motion image dataset includes: the target's motion trajectory image dataset and point cloud dataset.

[0036] Furthermore, the preprocessing specifically includes:

[0037] S11. Perform spatiotemporal alignment and format standardization on the target's point cloud dataset and motion trajectory image dataset;

[0038] S12. The point cloud data processed in step S11 is sequentially subjected to denoising, statistics, sampling and feature pre-calculation.

[0039] S13. For the motion trajectory image dataset processed in step S11, perform image enhancement, cropping and standardization processes in sequence.

[0040] The beneficial effects of the method described in this invention are as follows:

[0041] (1) The method described in this invention adopts a dual-tower deep learning architecture, which uses a point cloud branch + visual branch to perform multimodal feature fusion classification. PointNet++ is used to extract the geometric features of the target point cloud; ResNet is used to extract the visual features of the target, and then the point cloud and visual features are weighted and fused through an attention layer. Finally, the classification threshold is set by combining the density of the point target and the extended target to obtain the classification result. Through the above dual-tower deep learning architecture, the advantages of the two modalities are "strongly combined". For example, when there is a small target at a distance (the point cloud is sparse but the visual outline is distinguishable), the outline features are highlighted; when there is an occluded scene (the visual image is blurred but the point cloud is geometrically complete), the geometric features are highlighted, avoiding misjudgment by a single modality.

[0042] (2) The method described in this invention improves the SiamFC++ module by performing channel attention weighting (SE module) on shallow features, amplifying channels unique to small targets (such as channels with strong edge response), and suppressing background-dominated channels;

[0043] In template features, a mask is used to distinguish between the "target core region" and the "context region," which solves the problem of weak features for small targets and prevents background features from overshadowing the target.

[0044] After the cross-correlation matching of the template and the search features, the matching score is "spatially constrained". If the distance between a certain position and the historical target position exceeds a preset threshold (such as twice the target diameter), its matching score is reduced to avoid the model being interfered with by strong backgrounds in the distance.

[0045] A multi-resolution feature fusion mechanism and a multi-scale adaptive matching strategy are introduced into the search branch of the SiamFC++ module to address the fluctuations in the size of small targets.

[0046] (3) In the prediction branch of the BP-PMBM model, the method of the present invention replaces the near constant velocity module with the Transformer module. The Transformer module can solve the problem of "local dependency bias" by modeling global temporal correlation. The Transformer module can also learn multi-scale motion features in parallel, better adapt to complex scenes, and enhance motion interpretability by combining external environmental information.

[0047] (4) In the update branch of the BP-PMBM model, the method of the present invention replaces the BP module with the temporal BP module. The temporal BP module can handle the dynamic dependency structure formed by the change of the target over time, and is no longer limited to the fixed variable relationship in the static graph model. It can continuously and accurately update the motion trajectory of multiple targets. Attached Figure Description

[0048] Figure 1 This is a flowchart of the method described in this invention;

[0049] Figure 2 This is a flowchart illustrating the workflow of the PointNet++ and ResNet dual-tower classification structure described in this invention.

[0050] Figure 3 This is a schematic diagram illustrating the precision plots of OPE (Object Target Tracking) as described in this invention.

[0051] Figure 4 This is a schematic diagram illustrating the success plots of OPE (Object Target Tracking) as described in this invention.

[0052] Figure 5 This is a schematic diagram of the standard frame described in this invention;

[0053] Figure 6 This is a schematic diagram of the Siamfc++ block as described in this invention;

[0054] Figure 7 This is a schematic diagram of the improved Siamfc++ block described in this invention;

[0055] Figure 8This is a comparison chart of the tracking performance of the traditional BP module and the timing BP module described in this invention;

[0056] Figure 9 This is a comparison diagram of the position errors of the traditional BP module and the timing BP module described in this invention;

[0057] Figure 10 This is a comparison chart of the number of pixel changes between the traditional BP module and the timing BP module described in this invention;

[0058] Figure 11 This is a schematic diagram illustrating the accuracy of the traditional (static) BP module and the timing BP module described in this invention;

[0059] Figure 12 This is a schematic diagram illustrating the cumulative accuracy of the traditional BP module and the timing BP module described in this invention;

[0060] Figure 13 This is a comparison chart of the accuracy of the occluded frame and the normal frame described in this invention;

[0061] Figure 14 This is a comparison diagram of the trajectory tracking effects of the traditional BP module and the timing BP module described in this invention on single-target bending acceleration + sudden turning.

[0062] Figure 15 This is a schematic diagram illustrating the tracking performance of the BP-TPMB described in this invention under relatively simple target trajectories.

[0063] Figure 16 This is a schematic diagram illustrating the tracking performance of the BP-TPMB described in this invention under relatively complex target trajectories;

[0064] Figure 17 This is a schematic diagram illustrating the tracking effect of the TR-TBP-TPMB described in this invention under relatively complex target motion trajectories. Detailed Implementation

[0065] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This embodiment provides a hybrid tracking method for drone-captured targets based on multimodal classification. The flowchart of the method is as follows: Figure 1 As shown, the method includes the following steps:

[0067] S1. Collect motion trajectory image datasets and point cloud datasets of the target over a period of time using a drone equipped with LiDAR (Light Detection and Ranging), and preprocess the motion trajectory image datasets and point cloud datasets;

[0068] Point cloud data (such as LiDAR output) excels at characterizing the three-dimensional geometry of targets (position, curvature, and density, etc.), accurately distinguishing between "point targets" (such as small objects in the distance, with sparse point clouds and no obvious spatial extension) and "extended targets" (such as vehicles and pedestrians, with dense point clouds and complete spatial outlines). Visual data (such as camera images) excels at capturing the two-dimensional contours, textures, and other appearance features of targets, helping to determine whether a target has an "extended shape" (such as the rectangular outline of a vehicle or the humanoid outline of a pedestrian). Combining the two can compensate for the deficiencies of a single modality (such as information loss in point clouds when targets are occluded, and blurred outlines in vision under poor lighting conditions).

[0069] Accordingly, the method of the present invention collects motion trajectory image datasets and point cloud datasets of a target over a period of time using a LiDAR (Light Detection and Ranging) mounted on a UAV, and preprocesses the motion trajectory image datasets and point cloud datasets. Regarding the point cloud dataset, existing datasets can also be used, such as pseudo-3D point clouds generated by depth estimation models such as UAV123 and VisDrone2022, or UAV 3D Dataset can be directly converted into a real point cloud dataset for UAV 3D target tracking design.

[0070] Target motion trajectory image datasets can characterize the target's current velocity and position;

[0071] The preprocessing is as follows:

[0072] First, the input point cloud data and visual data (target motion trajectory image data) need to be spatiotemporally aligned and format-standardized to ensure modal consistency:

[0073] Spatiotemporal alignment: By using sensor calibration parameters (such as the extrinsic matrix of LiDAR and Camera), the point cloud coordinates are projected onto the (visual data) image pixel coordinate system, or vice versa, the image features are mapped onto the point cloud coordinate system to ensure that the spatial position of the same target corresponds in the two modalities; the difference in sensor sampling frequency (such as LiDAR 10Hz, Camera 30Hz) is resolved by using timestamp synchronization (such as interpolation completion).

[0074] Then, point cloud data preprocessing and visual data preprocessing are performed sequentially:

[0075] Point cloud data preprocessing:

[0076] Denoising: Statistical filtering and voxel filtering (reducing point cloud density and computational load) are used to denoise the point cloud dataset;

[0077] The statistics specifically include: removing nodes whose distance exceeds the neighboring point's range. The point cloud, in statistical filtering In the "Principles", It is the standard deviation of the distances from points in a local neighborhood of a point cloud to the center. Its core function is to quantify the "discreteness" of the local point cloud. The smaller the value, the more concentrated the points are in the neighborhood (the more regular the structure). The larger the value, the more dispersed the points within the neighborhood (potentially containing noise or edges). The statistical properties corresponding to a normal distribution are: approximately 99.7% of the valid data will fall within the mean. Points outside the specified range are identified as noise points and removed.

[0078] Sampling: To adapt to the input of PointNet++, the point cloud data is uniformly sampled to a fixed number (e.g., 2048 points), and FPS (farthest point sampling) is used to retain key geometric information;

[0079] Feature pre-computation: There may be problems such as noise, redundancy or scale inconsistency between the original features of point cloud data and the features extracted manually or by the network later. Therefore, in the preprocessing stage, the basic geometric features of the point cloud (such as the coordinates, normal vector and curvature of each point) are calculated in advance.

[0080] Preprocessing of visual data (target motion trajectory image dataset):

[0081] Image enhancement: Data enhancement techniques such as random flipping, brightness and contrast adjustment, and Gaussian blur are used to improve robustness;

[0082] Cropping: The specific location and range of the target in the image are determined by point cloud data, and then the original image is cropped accordingly, retaining the key area containing the target and removing redundant background, thereby reducing the interference of background information on subsequent processing and improving the accuracy and efficiency of target detection and recognition.

[0083] Standardization: Normalizing image pixel values ​​to... or It is compatible with ResNet input format.

[0084] S2. Input the preprocessed dataset into the PointNet++ and ResNet dual-tower classification structure, such as... Figure 2 As shown, the dual-tower classification structure divides the target (input data) into point targets and extended targets;

[0085] In existing UAV tracking methods, when tracking multiple targets, if point targets and extended targets coexist, the PMB filter, due to its measurement generation mechanism assuming that each target generates at most one measurement point, will misinterpret multiple measurement points generated by extended targets as multiple independent point targets, leading to errors such as overestimation of the number of targets and decreased tracking accuracy. Therefore, the method described in this invention introduces a recognition mechanism, using a point cloud-visual multimodal approach based on a PointNet++ and ResNet dual-tower deep learning architecture to classify targets.

[0086] The PointNet++ and ResNet dual-tower classification structure includes a PointNet++ point cloud branch and a ResNet vision branch;

[0087] PointNet++ is a classic architecture for point cloud feature extraction. Through a hierarchical structure of "sampling-grouping-feature aggregation", it can effectively extract local geometric features (such as curvature calculated by the difference in normal vectors of neighboring points, and local structural entropy quantified by the disorder of point cloud distribution), and is naturally adapted to the geometric distinction requirements of "point target vs. extended target".

[0088] The PointNet++ point cloud branch passes through the input layer, sampling layer, feature aggregation layer, global feature layer, and density feature calculation layer in sequence from input to output.

[0089] The workflow of PointNet++ point cloud branches is shown in Table 1:

[0090] Table 1

[0091]

[0092] The final output of the PointNet++ point cloud branch is a 257-dimensional point cloud feature vector. (256-dimensional geometric feature vector + 1-dimensional density feature → 257-dimensional point cloud feature vector), where the geometric features (such as curvature and local structural entropy) are extracted hierarchically from PointNet++, which can accurately depict the three-dimensional spatial structure of the target and is good at distinguishing between "point targets (sparse point cloud, no spatial extension)" and "extended targets (dense point cloud, with complete outline)". The density feature directly reflects the spatial extension of the target (density = total number of point clouds / target bounding box volume) and is a key auxiliary information for judging the target type.

[0093] CNN architectures such as ResNet have residual connections and deep feature extraction capabilities, and can automatically learn the contour features of the target (such as capturing edges and textures through convolutional layers and outputting contour correlations through fully connected layers). They have also proven their robustness in image classification tasks and are suitable for contour judgment in visual branches.

[0094] The ResNet vision branch goes through the input layer, residual layers (residual layers 1-4), global average pooling layer, first fully connected layer and second fully connected layer in sequence from input to output.

[0095] The workflow of the ResNet vision branch is shown in Table 2:

[0096] Table 2

[0097]

[0098] The final output of the ResNet visual branch is a 257-dimensional visual feature vector. (256-dimensional contour feature vector + 1-dimensional contour correlation score), where the contour features are extracted from deep layers of ResNet, which can capture the appearance information of the target such as two-dimensional edges and textures, and help determine whether the target has an "extended shape" (such as the rectangular contour of a vehicle); the contour correlation score (between 0 and 1) directly quantifies the completeness of the target contour and supplements the missing information of the point cloud in occluded scenes.

[0099] Point cloud density (reflecting the spatial extension of the target) and visual contour features (reflecting the completeness of the target's appearance) are strongly correlated. By using probabilistic fusion (such as weighted summation and attention mechanisms) and threshold judgment, classification accuracy can be effectively improved, avoiding misjudgment of a single modality (such as misjudging an extended target as a point target when the point cloud is sparse, or misjudging a point target as an extended target when the vision is blurred). Accordingly, the method described in this invention adopts a multimodal fusion module design to solve the final classification result. It adopts a fusion strategy of "feature splicing + attention weighting", that is, through an attention layer, it dynamically learns the weight allocation of the point cloud modality and the visual modality, and then completes the feature weighted fusion, making full use of the complementary information of the two modalities.

[0100] First, the final outputs of the PointNet++ point cloud branch and the ResNet visual branch are concatenated: the 257-dimensional (geometric) feature vector from the point cloud branch is concatenated with the 257-dimensional (contour) feature vector from the visual branch to obtain a 514-dimensional fused feature vector. The 514-dimensional fused feature vector simultaneously contains "three-dimensional geometric structure + spatial density" (point cloud advantage) and "two-dimensional contour + appearance integrity" (visual advantage), describing the target from different dimensions and achieving information complementarity that cannot be covered by a single modality (such as using visual contour to complete when point cloud is occluded, and using point cloud geometry correction when visual lighting is poor).

[0101] Next, attention weights are calculated through an attention layer: the attention layer learns the weights of the two modalities, allowing the more reliable modality to have a higher weight in the fusion process. (This addresses the modality imbalance problem, such as when the point cloud is clear but the visual image is blurry, the point cloud has a higher weight):

[0102] The attention layer passes through the following layers from input to output: the first fully connected layer, the first weight calculation module, the second fully connected layer, and the Softmax activation function.

[0103] The first fully connected layer receives a 514-dimensional fused feature vector and calculates the raw scores of point cloud features and visual features. and ;

[0104] and The first weight calculation module calculates the weights of the point cloud features and the visual features respectively: Let the weight of the point cloud data be... The weight of visual data is ,but , ,in, Represents natural numbers.

[0105] The first weight calculation module (multimodal fusion module) further fuses point cloud features and visual features according to weights to obtain the final fused features. Since the dimension after weighted summation is consistent with that of the unimodal model, therefore, latitude and and They are consistent, both being 257-dimensional.

[0106] Through the weighted fusion described above, the advantages of the two modalities are combined to create a powerful synergy. For example, when dealing with small targets at a distance (sparse point clouds but discernible visual outlines), Enhance and highlight contour features; in occluded scenes (visually blurred but with complete point cloud geometry), Enhance and highlight geometric features to avoid misjudgment of a single mode.

[0107] Final fusion features After passing through a second fully connected layer, the feature fusion scores of point cloud features and visual features are obtained. and ;

[0108] and After processing with the Softmax activation function, the output point target probability is... and extended target probability ,satisfy .

[0109] Classification threshold setting, basic threshold: such as Determined as an extended target, It is determined to be a point target;

[0110] Dynamic threshold adjustment: The threshold is optimized by combining point cloud density features (lowering the threshold for determining extended targets and point targets when the density is high, and raising the threshold for determining extended targets and point targets when the density is low). The calculation method for the threshold for determining extended targets is as follows: , This represents the maximum value of the extended target density. Describes the minimum value function;

[0111] The calculation method for the point target determination threshold is as follows:

[0112]

[0113] Output result: If Output "Extended Target", and then use the TR-TBP-TPMB algorithm. Output "point target", and then use the SiamFC++ improvement module. If neither of these conditions is met (fuzzy area): use the "majority voting" strategy (combining the classification results of the first 3 frames) or default to "extended target" (to avoid tracking loss caused by misjudgment of point target).

[0114] S3. Input the point target into the SiamFC++ improved module to obtain the point target tracking result;

[0115] Traditional Siamese networks (SiamFC++ modules) often use deep CNNs (such as AlexNet), which have high feature map downsampling rates (1 / 8 or 1 / 16). Small targets occupy only 1-2 pixels on the feature map, resulting in complete loss of detail. To improve the tracking accuracy of small targets, the method described in this invention improves SiamFC++ as follows:

[0116] In the feature extraction part of the SiamFC++ module, ResNet-18 is truncated to the conv3 layer, and an SE channel attention module is added after the conv3 layer;

[0117] Shallow feature priority: Lightweight CNNs (such as ResNet-18 and MobileNetV2) are used, and deep network layers are truncated to preserve high-resolution shallow features. Specifically:

[0118] Instead of using the conv4 / conv5 layers (downsampling by 16x) of ResNet-18, we use the output of the conv2 layer (downsampling by 4x) or the conv3 layer (downsampling by 8x) as the feature map.

[0119] Small targets can retain more than 4×4 pixels on the conv2 feature map (assuming the small target in the original image is 16×16 pixels, it becomes 4×4 after downsampling by 4 times), which is sufficient to express key information such as edges and textures.

[0120] Feature channel enhancement: Apply channel attention weighting to shallow features (e.g., using the SE module), amplify channels specific to small targets (e.g., channels with strong edge response), and suppress background-dominant channels.

[0121] Expand the size of the SiamFC++ module template area: Set the size of the SiamFC++ module template area to twice the size of the target box;

[0122] Context-aware template construction (addressing the issue of weak features in small targets)

[0123] Small targets have few features of their own, but the surrounding local context (such as a stable background and texture near the small target) can help distinguish the target from the background. Based on this, the method described in this invention is improved as follows:

[0124] Expanded template area: Traditional templates only include the area within the target bounding box. Instead, they include the target plus a local background 1-2 times the size of the surrounding area (e.g., if the target bounding box is 10×10, the template area is expanded to 20×20), integrating contextual features into the template.

[0125] Contextual attention mask: In template features, a mask is used to distinguish between the "target core region" and the "context region". The core region has a higher weight (e.g., 0.7) and the context region has a lower weight (e.g., 0.3) to prevent background features from overshadowing the target.

[0126] Focused search and matching mechanism (solving "background interference")

[0127] Small targets account for a very small percentage of the search area. If the search range is too large, background features will overwhelm target features. The improved method described in this invention applies a spatial constraint to the matching score during the target matching process in the SiamFC++ module. When the Euclidean distance between a candidate position and the center position of a historical target exceeds a preset threshold, the matching score of the candidate position is attenuated. The preset threshold is twice the target diameter.

[0128] Dynamically reduce the search area: The search area of ​​the traditional Siamese network is 5-8 times the size of the target. In small target scenarios, it is reduced to 3-4 times (e.g., for a target of 10×10, the search area is 30×30-40×40), reducing the proportion of background features.

[0129] Matching score correction: After the cross-correlation matching of the template and the search features, the matching score is "spatially constrained". If the distance between a certain position and the historical target position exceeds a preset threshold (such as twice the target diameter), its score is reduced to avoid the model being interfered with by strong backgrounds in the distance.

[0130] Small targets may exhibit slight scale variations (e.g., becoming smaller at long distances and larger at close distances), making it difficult for single-scale features to adapt. Therefore, the improved scheme of the method described in this invention is to introduce a multi-resolution feature fusion mechanism and a multi-scale adaptive matching strategy into the search branch of the SiamFC++ module.

[0131] The multi-resolution feature fusion mechanism specifically involves upsampling low-resolution features to restore them to a high-resolution scale, and then stitching them together with other high-resolution features.

[0132] The multi-scale adaptive matching strategy is as follows: In the current frame, several scale versions of the template features are generated, and each scale template is matched with the fusion features of the corresponding scale in the search area. Finally, the scale version with the highest matching score is selected from the several scale matching results as the target localization result of the current frame.

[0133] Multi-resolution feature fusion: In the search branch, features of conv2 (1 / 4 downsampling) and conv3 (1 / 8 downsampling) are extracted simultaneously. Low-resolution features are then concatenated with high-resolution features through upsampling, thus fusing details and semantics at different scales.

[0134] Scale-adaptive matching: Generate 2-3 scales (e.g., 1.0×, 1.2×, 0.8×) for the template features, and match them with the corresponding scale features of the search region. Take the highest score as the final result to deal with the scale fluctuations of small targets.

[0135] The specific implementation methods of the above improvements are as follows:

[0136] Network structure:

[0137] Feature extractor: ResNet-18 is used, truncated to the conv3 layer (outputting an 8x downsampled feature map), and an SE channel attention module is added after conv3.

[0138] Template branch: Input the extended target region of the first frame (target + 1 times the context), output 128-channel template features (15×15 in size, corresponding to the size of the small target on the feature map).

[0139] Search branch: Input a region three times the size of the target in the current frame, and output a 128-channel search feature (45×45 in size).

[0140] Matching process:

[0141] Template features and search features are cross-correlated to calculate similarity (similar to convolution), resulting in a 45×45 response map (higher values ​​indicate greater similarity).

[0142] Spatial constraints (suppressing distant locations) and scale fusion (combining responses at different scales) are applied to the response map, and the location of the maximum value is taken as the target center to regress the bounding box.

[0143] Training techniques:

[0144] Datasets: Train on small target-specific datasets (such as the small target subset in UAV123 and the VisDrone dataset) to enhance the model's sensitivity to small targets.

[0145] Loss function: In addition to the traditional classification loss (distinguishing between target and background), a "center offset penalty" is added. If the center of the predicted box deviates from the true center by more than the radius of the small target, the loss is increased to force the model to locate accurately.

[0146] Mathematical modeling process:

[0147] High-resolution feature extraction and enhancement

[0148] For small targets, the feature extraction network was changed from deep layers (conv4 / conv5) in the standard SiamFC++ to shallow layers (conv2 / conv3), and the downsampling rate was reduced from 16 to 4 to ensure the size of the feature map for small targets. ;

[0149] Add channel attention (SE module) to enhance key features of small targets (such as edge channels).

[0150] Template context blending and masking

[0151] The method described in this invention addresses the core pain points of small target tracking through "high-resolution feature preservation + context enhancement + focused matching," achieving an excellent balance between accuracy and speed. It is currently the most practical and preferred solution for small target tracking scenarios.

[0152] The template area is expanded to "target + context" (the size of the template area is 1.5-2 times the size of the target box), and a mask is used to distinguish the core and context areas:

[0153] ,in, Represents the pixel coordinates (or spatial location indexes of the feature map) on the template feature map. Represents the horizontal (width) coordinate index of the template feature map, corresponding to the column dimension of the feature map; This represents the vertical (height) coordinate index of the template feature map, corresponding to the row dimension of the feature map; in this embodiment, This means that the core area has a higher weight.

[0154] Focused cross-correlation matching

[0155] The search area has been narrowed down to (The original standard was) ), Search feature map size Cross-correlation response diagram range Restricted: ,in, Represents the correlation function. This represents the template feature map after being weighted by the "target core region / context region" mask. Indicates the narrowed search area The corresponding search feature map, and These represent the horizontal and vertical position indices of the search feature map, respectively;

[0156] Add spatial constraints (suppress long-distance positions): ,in Indicates the central location of the historical target. Represents Euclidean distance. This represents the penalty coefficient, which suppresses responses that are far removed from historical positions.

[0157] Small goal fit for classification and regression

[0158] The classification branch incorporates a "center offset penalty," making it more sensitive to center positioning errors of small targets. ,in, Indicates the true center of the target. In this embodiment, the penalty weight is represented. The value is 0.4. Represents the template feature map or search feature map The true category label corresponding to the location is used to indicate whether the location belongs to "target" or "background". It is usually a binary discrete value. If the location falls within the true bounding box of the small target (belonging to the target's core area or effective target region), then... If it falls within the background area (not the target area), then ; The model predicts the probability, and the model predicts the feature map through the classification branch. The probability that a location belongs to the "target class" is a continuous value (ranging from 1 to 1). In the SiamFC++ improved module, after obtaining the response map through cross-correlation of template features and search features, the classification branch will process each element in the response map... The response values ​​at a location are probability-mapped (e.g., using the Sigmoid function), and the final output is... (That is, the confidence level of the target at this position).

[0159] The regression branch incorporates a "scale constraint" to prevent excessive shrinkage of the bounding box for small targets. ,in, This represents a loss function that combines the advantages of L1 and L2 loss functions, used to reduce gradient explosion and convergence difficulties. This represents the model's predicted bounding box, indicating the predicted location and scale of small objects in the current frame, output by improving the regression branch of SiamFC++. Essentially, it's a vector containing four key parameters (typically in the format of...). The horizontal and vertical coordinates represent the width and height of the bounding box, respectively. This represents the actual bounding box. This represents the minimum size threshold, in this embodiment... The value is 4. This indicates the width of the prediction box. This indicates the height of the prediction box.

[0160] S4. Extend the target input to the TR-TBP-TPMB model to obtain the extended target tracking results;

[0161] The TR-TBP-TPMB model was obtained by improving the BP-PMBM model.

[0162] S41. In the prediction branch of the BP-PMBM model, replace the near-constant velocity module with the Transformer module.

[0163] The BP-TMPB model, which employs a "near-constant velocity model," assumes that the target's velocity changes little (approximately constant). However, when the target undergoes complex maneuvers such as rapid acceleration, deceleration, or sharp turns, this model cannot accurately predict drastic changes in the target's motion state, leading to problems such as increased tracking errors and easy target loss. Therefore, it is not adaptable to complex maneuvers. The method described in this invention improves the BP-PMBM model as follows: In the prediction branch of the BP-PMBM model, the near-constant velocity module is replaced with a Transformer module. The Transformer module captures temporal global dependencies through a self-attention mechanism and improves the ability to model long sequences through parallel processing.

[0164] The Transformer module overcomes the reliance of near-constant velocity models on the premise of "small speed changes." Through a self-attention mechanism, it simultaneously monitors different historical states over a longer timeframe, capturing both long-term trends and short-term abrupt changes in target motion. This results in greater adaptability to complex maneuvers such as acceleration, deceleration, and sharp turns. Compared to near-constant velocity models that can only extrapolate based on linear state transitions, the Transformer module automatically learns multi-scale motion features while processing historical state sequences in parallel. It dynamically focuses on the most critical moments of the current motion state using attention weights, thus enabling it to model nonlinear motion patterns. Furthermore, its input can be expanded to a combination of target state and external environmental information, allowing the model to automatically adjust its predictions of future motion based on factors such as road structure and scene changes. This makes the trajectory prediction of extended targets more interpretable and environmentally adaptable. Finally, the temporal encoding output by the Transformer module replaces traditional linear predictions, enabling the method described in this invention to maintain more stable tracking accuracy and robustness in dynamically changing scenarios.

[0165] In summary, the advantages of using Transformer are:

[0166] 1. Global temporal correlation modeling can solve the problem of "local dependency bias";

[0167] 2. Parallel learning of multi-scale motion features to adapt to complex scenarios: The added attention mechanism can simultaneously focus on motion patterns at different time scales (such as "short-term acceleration changes" and "long-term driving direction").

[0168] 3. Enhance motion interpretability by incorporating external environmental information: The input to the Transformer module can be expanded to "target state + environmental features" (such as road curvature, traffic light status), and the influence of the environment on motion is integrated through a cross-attention mechanism. For example, if a "red light at the intersection ahead" is detected, the Transformer module will adjust the prediction weights for target deceleration based on environmental features.

[0169] The applications of the near-constant speed module and the Transformer module in the method described in this invention are shown in Table 3:

[0170] Table 3

[0171]

[0172] S42. In the update branch of the BP-PMBM model, replace the BP module with the time-series BP module.

[0173] Traditional backpropagation (BP) is suitable for static graph models (where variables and dependencies are fixed). However, in dynamic scenarios (such as time-varying target tracking and real-time sensor fusion), the graph structure or variables are updated over time. Therefore, the method described in this invention uses temporal BP to improve upon it. Temporal BP (Temporal Belief Propagation) is a modeling method that extends static belief propagation (BP) to dynamic temporal scenarios. Its core is to construct a temporal factor graph to describe the dependencies of variables over time and define cross-time message passing rules to achieve online reasoning about the state of dynamic systems.

[0174] The temporal BP module can handle the dynamic dependency structure formed by the target changing over time, and is no longer limited to the fixed variable relationships in static graph models. Accordingly, the method described in this invention replaces the traditional BP module with a temporal BP module in the update branch of the BP-PMBM model. Traditional BP is suitable for static graph models (variables and dependencies are fixed), but the trajectory in the extended target tracking process changes at every moment, causing the traditional BP module to be unable to continuously and accurately update the motion trajectory of multiple targets. The temporal BP module constructs a factor graph that unfolds across time, explicitly representing the relationship between trajectory state, observation, and associated variables as a temporal chain structure. It achieves state prediction through forward propagation and smooths the correction of historical trajectories through backpropagation, making the inference process both linear and temporally consistent.

[0175] Forward messages are responsible for transmitting the trajectory confidence from the previous moment to the current moment, and combining the observation results to form predictions and updates; backward messages, after obtaining future correlations and observations, reverse-correct past trajectories to make the final estimate more stable and more in line with the time pattern of the actual trajectory evolution.

[0176] Since the temporal BP module retains the mixed structure of Poisson and Dobernoli in the BP-PMBM model and introduces intensity and trajectory transfer factors that are recursively applied over time, it can naturally express the assumptions of trajectory generation, extinction, undetected continuation, and cross-time correlation. As a result, it can achieve higher tracking accuracy and stronger robustness than the traditional BP module in complex temporal environments with multiple targets.

[0177] In this embodiment, the prediction and update process of the BP-PMBM model is extended temporally to make it applicable to multi-target tracking scenarios. Specifically, the temporal prediction step is used to recursively infer the trajectory state, the intensity of undetected targets, and the probability of new targets appearing between adjacent time points; the temporal update step uses a temporal belief propagation mechanism to jointly infer observation information from multiple time points, thereby obtaining a temporally consistent target state estimate.

[0178] In the temporal prediction step, the survival probability of Bernoulli components existing in the previous time step is recursively extrapolated. The survival probability is determined by the terminal state of the component in the previous time step, and the predicted distribution of the trajectory at the current time step is obtained based on the motion transfer model. Simultaneously, newly emerging targets that may appear in the current time step are modeled using a Poisson point process to predict their intensity distribution. Furthermore, for potential targets that have not been observed or detected, their intensity is also recursively extrapolated over time using a Poisson process to maintain the ability to model the continuity of undetected trajectories.

[0179] In the temporal update step, the predicted PMBM density is first used as the initial input for the forward message. Then, forward message propagation is performed sequentially along the time series, continuously updating the forward confidence score based on the temporal transition relationships of the trajectory states and the compatibility information of the observation factors. After completing forward inference, smoothing message propagation is performed in reverse time, using observations and correlation information from future moments to correct the historical trajectory states, obtaining a smoothed post-confidence score that integrates forward and backward information. Through this bidirectional temporal belief propagation mechanism, a unified update of the dynamic multi-objective state is achieved.

[0180] After obtaining the smoothed PMBM density, in order to reduce computational complexity and facilitate recursive processing in subsequent time steps, the density is transformed into a more simplified PMB approximation form by minimizing the divergence, thereby improving computational feasibility while ensuring estimation accuracy.

[0181] The application of the BP module and the time-series BP model in the method described in this invention is shown in Table 4:

[0182] Table 4

[0183]

[0184] Example 2

[0185] This embodiment further defines Embodiment 1. In the point target tracking branch, comparative experiments were conducted using the SiamFC++ module, the SiamMRC module, and the improved SiamFC++ module, respectively. The results of the comparative experiments are as follows: Figure 3 and 4 As shown,

[0186] Figure 3 This demonstrates the change in point target trajectory tracking accuracy (OPE) as the location error threshold increases (to accommodate the unavoidable positioning deviations of point targets under conditions of noise interference, weak texture, and long-term temporal drift, the threshold for successful tracking needs to be gradually relaxed). Figure 3 It can be seen that the improved SiamFC++ module has significantly higher point target trajectory tracking accuracy than the SiamFC++ module and the SiamMRC module as the position error threshold continues to increase.

[0187] Figure 4 The diagram illustrates how the point target trajectory tracking success rate (OPE) changes as the overlap threshold increases (to verify the overlap quality between the tracking result and the true trajectory under stricter spatial consistency requirements, the overlap threshold for determining success needs to be gradually increased). Figure 4 It can be seen that, although the success rate of point target trajectory tracking of the improved SiamFC++ module decreased as the overlap threshold continued to increase, it was still significantly higher than that of the SiamFC++ module and the SiamMRC module.

[0188] As shown in Table 5, this embodiment also selected the tracking coordinate data of the SiamFC++ module and the improved SiamFC++ module at frames 000001, 000010, 000100, 001000, and 003000. The data in Table 5 shows that in each frame, the tracking result of the improved SiamFC++ module (improved SiamFC++ bounding box coordinates) is closer to the standard bounding box coordinates than the tracking result of the SiamFC++ module (SiamFC++ bounding box coordinates). That is, the tracking performance of the improved SiamFC++ module is better than that of the SiamFC++ module. For example: Figure 5 , 6 Figures 7 and 8 respectively show schematic diagrams of standard frame coordinates, Siamfc++ frame coordinates, and improved Siamfc++ frame coordinates under frame 000100.

[0189] Table 5

[0190]

[0191] Example 3

[0192] This embodiment further defines Embodiment 1. In this embodiment, the target tracking branch is extended, and comparative experiments are conducted using both a traditional BP module and a temporal BP module. This embodiment selects a sequence of 5 frames of 2×2 moving blocks on an 8×8 grid, adds mild Gaussian noise as input, and verifies the tracking results of both the traditional BP module and the temporal BP module in noisy environments. Figure 8 As shown, compared with the traditional BP module, the time-based BP module has better positioning accuracy and can correct deviations (remove noise) during tracking, while the positioning error of the traditional BP module accumulates with frame sequence (more affected by noise). Meanwhile, as... Figure 9 and 10 As shown, the time-series BP module has stronger inter-frame stability and smoother, more continuous block trajectories, while the traditional BP module exhibits more drastic inter-frame changes.

[0193] Therefore, the tracking performance of the traditional (static) BP module and the temporal BP module in complex occlusion scenarios was tested, such as... Figure 11 As shown, in the first occlusion frame (the occlusion frame between the 2nd and 4th frames), the accuracy of the traditional BP module drops directly to 0, while the timing BP module still maintains a certain accuracy. In the second occlusion frame (the 6th frame), although the accuracy of the timing BP module also drops to 0, it immediately recovers to a certain accuracy in the next frame, while the accuracy of the traditional BP module remains at 0.

[0194] like Figure 12 As shown, this embodiment selects the cumulative average accuracy of the traditional BP module and the time-series BP module on 8 targets, from... Figure 12 It can be seen that the cumulative average accuracy of the timing BP module is significantly higher than that of the traditional BP module, and the long-term tracking consistency of the timing BP module is far superior to that of the traditional BP module.

[0195] like Figure 13 As shown, the timing BP module achieves higher tracking accuracy for extended targets in both occluded and normal frames compared to the traditional BP module.

[0196] A comparison of the trajectory tracking performance of the traditional BP module and the time-series BP module in single-target bending acceleration + sudden turning is as follows: Figure 14 As shown, by Figure 14 It can be seen that the temporal BP module closely approximates the target's true trajectory during the entire process of bending acceleration and sudden turns. However, the traditional BP module completely loses the target during sudden turns (bending point failure). That is, by introducing temporal constraints, the temporal BP module significantly outperforms the traditional BP module, which relies solely on single-frame information, in terms of occlusion robustness, inter-frame consistency, and final positioning accuracy.

[0197] Example 4

[0198] This embodiment further defines embodiment 1, such as... Figure 15 As shown, even with relatively simple target trajectories, BP-TPMB still misclassifies clutter as new targets (predicted extended target range). These false targets (excessive predicted extended target range) constitute noise in the extended target tracking results.

[0199] like Figure 16 As shown, BP-TPMB exhibits significant errors when dealing with complex target trajectories (e.g., a noticeably longer trajectory due to increased target acceleration per unit time), as it uses a near-constant velocity model and cannot handle complex situations such as sudden stops and starts. The NCV model also shows considerable tracking errors for maneuvering targets (e.g., sudden turns or speed changes).

[0200] The tracking performance of TR-TBP-TPMB under relatively complex target motion trajectories, such as Figure 17 As shown, TR-TBP-TPMB effectively avoids the problem of point targets forming clutter and being mistakenly estimated as extended targets by classifying point targets and extended targets at the very beginning.

[0201] Furthermore, by introducing the Transformer module and the temporal BP module to capture nonlinear correlations in historical trajectories (such as long-term velocity change trends), the accuracy of state prediction in complex scenarios is improved.

Claims

1. A hybrid tracking method for targets captured by unmanned aerial vehicles (UAVs) based on multimodal classification, characterized in that, The method includes the following steps: S1. Collect motion image datasets of the target using a drone equipped with a lidar, and preprocess the datasets. S2. Input the preprocessed dataset into the PointNet++ and ResNet dual-tower classification structure. The dual-tower classification structure classifies the target into point targets and extended targets. The PointNet++ and ResNet dual-tower classification architecture includes a PointNet++ point cloud branch, a ResNet vision branch, and an attention layer; The PointNet++ point cloud branch passes through the sampling layer, feature aggregation layer, global feature layer, and density feature calculation layer sequentially from input to output. The ResNet visual branch passes through the residual layer, the global average pooling layer, the first fully connected layer, and the second fully connected layer in sequence from input to output. The attention layer passes through the following layers from input to output: the first fully connected layer, the first weight calculation module, the second fully connected layer, and the Softmax activation function. S3. Input the point target into the SiamFC++ improved module to obtain the point target tracking result; S4. Extend the target input to the TR-TBP-TPMB model to obtain the extended target tracking results.

2. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 1, characterized in that, The TR-TBP-TPMB model is obtained by improving the BP-PMBM model. The improvement is as follows: S41. In the prediction branch of the BP-PMBM model, replace the near-constant velocity module with the Transformer module. S42. In the update branch of the BP-PMBM model, replace the BP module with the time-series BP module.

3. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 2, characterized in that, The SiamFC++ improvement module specifically includes: In the feature extraction part of the SiamFC++ module, ResNet-18 is truncated to the conv3 layer, and an SE channel attention module is added after the conv3 layer; Expand the size of the SiamFC++ module template area: Set the size of the SiamFC++ module template area to twice the size of the target box; In the target matching process of the SiamFC++ module, spatial constraints are imposed on the matching score; A multi-resolution feature fusion mechanism and a multi-scale adaptive matching strategy are introduced into the search branch of the SiamFC++ module.

4. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 3, characterized in that, The spatial constraint applied to the matching score during the target matching process in the SiamFC++ module is as follows: when the Euclidean distance between the candidate position and the historical target center position exceeds a preset threshold, the matching score of the candidate position is attenuated. The preset threshold value is twice the target diameter.

5. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 4, characterized in that, The multi-resolution feature fusion mechanism specifically involves upsampling low-resolution features to restore them to a high-resolution scale, and then stitching them together with other high-resolution features.

6. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 5, characterized in that, The multi-scale adaptive matching strategy is as follows: In the current frame, several scale versions of the template features are generated, and each scale template is matched with the fusion features of the corresponding scale in the search area. Finally, the scale version with the highest matching score is selected from the several scale matching results as the target localization result of the current frame.

7. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 6, characterized in that, The target motion image dataset includes: the target motion trajectory image dataset and the point cloud dataset.

8. The hybrid tracking method for UAV-captured targets based on multimodal classification according to claim 7, characterized in that, The preprocessing specifically includes: S11. Perform spatiotemporal alignment and format standardization on the target's point cloud dataset and motion trajectory image dataset; S12. The point cloud data processed in step S11 is sequentially subjected to denoising, statistics, sampling and feature pre-calculation. S13. For the motion trajectory image dataset processed in step S11, perform image enhancement, cropping and standardization processes in sequence.

Images