Remote sensing video multi-target tracking method and system based on detection association combination

Abstract

The invention provides a remote sensing video multi-target tracking method and system based on detection association combination, and relates to the technical field of computer vision and multi-target tracking, and the method comprises the following steps: collecting remote sensing data, and carrying out the data marking and preprocessing operation, and obtaining a standardized sample set; constructing a detection association joint model, and training the detection association joint model based on the standardized sample set to obtain a trained detection association joint model; and inputting to-be-detected remote sensing data, and performing multi-target tracking reasoning on the to-be-detected remote sensing data by using the trained detection association joint model to obtain a multi-target tracking result. According to the method, the core pain points of high calculation redundancy, large positioning ambiguity and poor association accuracy in remote sensing video multi-target tracking are effectively solved, and collaborative improvement of precision and efficiency is realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multi-target tracking technology, and in particular to a method and system for multi-target tracking of remote sensing video based on detection association. Background Technology

[0002] In recent years, with the rapid development of satellite and UAV remote sensing technologies, remote sensing video data has become an important means of acquiring dynamic information in fields such as agricultural monitoring, land surveys, and disaster assessment. Multi-target tracking technology, as a core component of remote sensing video analysis, aims to continuously locate multiple targets of interest in a continuous image sequence while maintaining their identity consistency across frames. Currently, the mainstream method based on a two-stage "detection-association" framework first detects targets independently in each frame, and then connects the detection boxes in different frames to form a trajectory using data association algorithms.

[0003] Despite significant progress made in this type of method, a series of key technical bottlenecks remain when dealing with high-resolution, large-scale, densely packed, and environmentally complex remote sensing videos. These bottlenecks are specifically manifested in the following ways: First, computational redundancy is high, and real-time performance is difficult to guarantee. Traditional two-stage tracking methods typically employ independent detection and association models. The detection module (such as anchor-box based Faster R-CNN and YOLO series) needs to perform intensive feature extraction on the input image to complete localization and classification, while the association module (such as SORT and DeepSORT) often needs to extract features from the image region or the detected target again for appearance matching or motion prediction. This pattern of repeated feature extraction leads to huge computational overhead, making it difficult to meet the urgent needs of satellite downlink or UAV airborne platforms for real-time processing capabilities.

[0004] Secondly, target localization suffers from significant ambiguity, limiting detection accuracy. Existing detectors generally rely on pre-set anchor boxes for target regression. However, targets in remote sensing scenarios (such as irregularly shaped farmland, agricultural machinery, and disaster-stricken buildings) vary greatly in shape, size, and orientation, resulting in low matching accuracy between pre-set anchor boxes and the true target boundaries, thus causing serious localization ambiguity. Especially in situations with dense targets, occlusion, or blurred edges (such as cloud cover or shadows), the anchor box mechanism is prone to issues such as missed detections, false detections, and misfitting bounding boxes, directly affecting the quality of subsequent correlation inputs.

[0005] Finally, the association accuracy is poor and the trajectory stability is insufficient. Current association strategies mostly rely on simple intersection-union ratios or similarity calculations combined with shallow appearance features. In complex remote sensing scenarios, when multiple targets have similar appearances (e.g., multiple agricultural machines of the same model), their trajectories intersect, or temporary full occlusion occurs, such single or shallow association features are prone to causing incorrect matching or ID switching, disrupting the continuity and consistency of the trajectory. Furthermore, the independent optimization of detection and association tasks in the tracking framework can easily lead to conflict between task objectives and mismatch in feature representation, further weakening the robustness of the overall tracking.

[0006] How to solve the above-mentioned technical problems is the challenge facing this invention. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a detection-association-based multi-target tracking method and system that effectively solves the core pain points of high computational redundancy, large positioning ambiguity, and poor correlation accuracy in remote sensing video multi-target tracking, achieving a synergistic improvement in accuracy and efficiency.

[0008] The technical solution adopted by this invention to solve its technical problem is as follows: This invention provides a method for multi-target tracking in remote sensing video based on detection association, comprising the following steps: Remote sensing data is collected and labeled and preprocessed to obtain a standardized sample set; Construct a detection association joint model, train the detection association joint model based on a standardized sample set, and obtain the trained detection association joint model; Input the remote sensing data to be detected, and use the trained detection association joint model to perform multi-target tracking inference on the remote sensing data to be detected, and obtain the multi-target tracking results.

[0009] Preferably, the data annotation includes using a semi-automatic annotation method to annotate the key point coordinates of each target in the remote sensing data, and assigning a unique tracking ID to each target; The key point coordinates include the center point coordinates and corner point coordinates of each target.

[0010] Preferably, the preprocessing includes noise reduction, pixel value normalization, size unification, illumination compensation, and timing standardization.

[0011] Preferably, the detection association joint model includes a feature extraction module, a dual attention detection module, and a multi-dimensional association module; The feature extraction module uses a ResNet50 network to extract features from a standardized sample set and output a multi-scale spatial feature map. The dual attention detection module is used to generate center point heatmaps and corner point heatmaps based on multi-scale spatial feature maps, determine the center point coordinates and corner point coordinates of the target based on the center point heatmaps and corner point heatmaps, and generate the target bounding box based on the center point coordinates and corner point coordinates of the target. The multi-dimensional association module is used to extract the geometric and appearance features of the target bounding box; generate a relation matrix based on the geometric and appearance features; infer the relation matrix through a GRU network to obtain the relation matrix matching probability; determine whether the target matches based on the relation matrix matching probability to obtain the target matching result; and associate the target in different frames based on the target matching result.

[0012] Preferably, the extraction of geometric and appearance features of the target bounding box includes extracting geometric features by calculating the IoU and Euclidean distance between the target bounding box of the current frame and the target bounding box of the historical frames; and extracting appearance features by calculating the cosine similarity between the target bounding box of the current frame and the target bounding box of the historical frames.

[0013] Preferably, the step of determining whether the target matches based on the matching probability of the relation matrix includes setting a threshold for the matching probability of the relation matrix. If the matching probability of the relation matrix is ​​greater than or equal to the threshold, the target is determined to match; if the matching probability of several relation matrices is less than the threshold, the target is determined not to match.

[0014] Preferably, the training detection association joint model uses a hierarchical joint loss function combined with backpropagation and gradient descent algorithms to update the model parameters; the hierarchical joint loss function includes keypoint localization focus loss, bounding box regression loss, and association matching focus loss; the keypoint localization focus loss formula is expressed as follows:

[0015] in, For the sample size, The heatmap results are obtained based on the true values. This is the heatmap response obtained through keypoint detection. To locate the hyperparameters of the focus loss for key points; The bounding box regression loss formula is expressed as follows:

[0016] in, These are the weight parameters for the bounding box regression loss. These are the true values ​​for the detection box size and offset, respectively. These are the dimensions and offsets obtained by the key point detection module, respectively. The formula for the association matching focus loss is expressed as follows:

[0017] in, For the true labels of the relationship, The model predicts the matching probability. The weight parameters for the association matching focus loss, This refers to the hyperparameters of the association matching focus loss.

[0018] This invention also provides a remote sensing video multi-target tracking system based on detection association joint method, comprising: The data acquisition and preprocessing module is used to acquire remote sensing data and perform data annotation and preprocessing operations to obtain a standardized sample set. The model building and training module is used to build a detection association joint model, train the detection association joint model based on a standardized sample set, and obtain the trained detection association joint model. The multi-target tracking execution module is used to input the remote sensing data to be detected, and to use the trained detection association joint model to perform multi-target tracking inference on the remote sensing data to be detected, so as to obtain the multi-target tracking results.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described remote sensing video multi-target tracking method based on detection association.

[0020] The present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described remote sensing video multi-target tracking method based on detection association.

[0021] The beneficial effects of this invention are as follows: It effectively solves the core pain points of high computational redundancy, large localization ambiguity, and poor association accuracy in multi-target tracking of remote sensing videos, achieving a synergistic improvement in accuracy and efficiency. By sharing the common features extracted by the ResNet50 backbone network and designing a hierarchical joint loss function for collaborative optimization, it fundamentally eliminates the redundant feature extraction and computational redundancy caused by the independent training and inference of the detection and association modules in traditional two-stage methods. A "center point-corner point" dual attention module is adopted to directly regress the key point coordinates of the target to generate the bounding box. This design effectively overcomes the localization ambiguity caused by irregular shapes and blurred edges of remote sensing targets, significantly improving the detection accuracy of small-sized targets in occluded and blurred scenes. By fusing geometric features (IoU, center point distance) and appearance features (cosine similarity) to construct a multi-dimensional relationship matrix, and then using a GRU network to infer the matching probability, this method can maintain high association accuracy even in complex scenes where targets have similar appearances (such as multiple agricultural machines of the same model) or temporary occlusion, and significantly reduces the number of ID switching. By employing a hierarchical joint loss function, the model training bias caused by the extreme imbalance of positive and negative samples in the detection task and the extremely low proportion of matching samples in the association task is specifically addressed. This enables the model to optimize the detection and association sub-tasks in a balanced manner, effectively improving the training convergence speed and model accuracy. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating the method steps of the present invention.

[0023] Figure 2 This is a system module diagram of the present invention.

[0024] Figure 3 This is a schematic diagram of the technical logic of the present invention.

[0025] Figure 4 This is a schematic diagram showing the detection comparison results of deepsort and DAUL-MOT on the VisDrone2019-MOT dataset in Embodiment 3 of the present invention.

[0026] Figure 5 This is a schematic diagram comparing the tracking performance of deepsort and DAUL-MOT in the 70th frame of the uav0000126_00001_v video sequence of the VisDrone2019-MOT dataset according to Embodiment 3 of the present invention.

[0027] Figure 6 This is a schematic diagram comparing the tracking performance of deepsort and DAUL-MOT in the 139th frame of the uav0000126_00001_v video sequence of the VisDrone2019-MOT dataset according to Embodiment 3 of the present invention.

[0028] Figure 7 This is a schematic diagram comparing the tracking performance of deepsort and DAUL-MOT in the 180th frame of the uav0000126_00001_v video sequence of the VisDrone2019-MOT dataset according to Embodiment 3 of the present invention.

[0029] Figure 8 This is a schematic diagram of the internal structure of Embodiment 4 of the present invention. Detailed Implementation

[0030] To clearly illustrate the technical features of this solution, the following detailed implementation method will be used to explain the solution.

[0031] Example 1: See Figure 1 , Figure 3 As shown, this embodiment is a remote sensing video multi-target tracking method based on detection association joint method, including the following steps: S1. Collect remote sensing data and perform data annotation and preprocessing operations to obtain a standardized sample set; Data annotation includes using a semi-automated annotation method to annotate the key point coordinates of each target in the remote sensing data and assigning a unique tracking ID to each target; the key point coordinates include the center point coordinates and corner point coordinates of each target.

[0032] Preprocessing includes noise reduction, pixel value normalization, size unification, illumination compensation, and timing standardization.

[0033] It should be noted that, using the original remote sensing video set as input, the LabelStudio annotation platform was used for semi-automated annotation, balancing accuracy and efficiency: the annotation process adopted an "AI pre-annotation + manual correction" mode. First, the initial annotation boxes were automatically generated using a pre-trained RT-DETRv3 model, and manual correction was only performed on frames with annotation errors > 5 pixels. In view of the irregular characteristics of remote sensing targets, the coordinates of the "center point + 4 corner points" key points of each target need to be annotated, and a unique tracking ID is assigned to the same target in the video frame to ensure cross-frame target correlation. At the same time, in order to supplement the scarce samples such as occlusion and small targets in the real data, the VideoLDM video diffusion model can be used to synthesize target videos such as smoke and moving agricultural machinery in "urban, forest and disaster" scenarios, and the ratio of synthetic data to real data is controlled at 1:3, finally forming an annotated video set containing key points and tracking IDs.

[0034] During preprocessing, a 3×3 kernel Gaussian filtering algorithm is first used to remove salt-and-pepper noise caused by electromagnetic interference during satellite and UAV transmission. Then, the pixel values ​​of the video frames are normalized to the [0,1] interval and uniformly resized to 1024×1024 size to balance resolution and computational efficiency. Subsequently, illumination compensation is achieved through gamma correction with γ=1.2 to solve the brightness difference problem between shadow areas and bright areas in backlit scenes. Finally, time-series frame extraction is performed at a frame rate of 30fps, and interpolation is used to supplement the original 15fps video to form a continuous frame sequence to preserve the temporal correlation between frames, and finally, the preprocessed continuous frame annotation data is obtained.

[0035] Finally, based on the preprocessed continuous frame labeled data, it is divided into training set, validation set and test set in a ratio of 7:2:1—the training set is used for model parameter learning, the validation set is used for hyperparameter tuning, and the test set is used for final performance verification.

[0036] S2. Construct a joint detection association model. Train the joint detection association model based on a standardized sample set to obtain the trained joint detection association model. The joint model for detecting associations includes a feature extraction module, a dual-attention detection module, and a multi-dimensional association module. The feature extraction module uses a ResNet50 network to extract features from a standardized sample set and output multi-scale spatial feature maps. The dual attention detection module is used to generate center point heatmaps and corner point heatmaps based on multi-scale spatial feature maps, determine the center point coordinates and corner point coordinates of the target based on the center point heatmaps and corner point heatmaps, and generate the target bounding box based on the center point coordinates and corner point coordinates of the target. The multi-dimensional association module is used to extract the geometric and appearance features of the target bounding box; a relation matrix is ​​generated based on the geometric and appearance features; the relation matrix is ​​inferred through the GRU network to obtain the relation matrix matching probability; the target is judged to match based on the relation matrix matching probability to obtain the target matching result; and the targets in different frames are associated based on the target matching result.

[0037] Extracting geometric and appearance features of the target bounding box includes extracting geometric features by calculating the IoU and Euclidean distance between the target bounding box in the current frame and the target bounding boxes in historical frames; and extracting appearance features by calculating the cosine similarity between the target bounding box in the current frame and the target bounding boxes in historical frames.

[0038] Determining whether a target matches based on the matching probability of the relation matrix involves setting a threshold for the matching probability of the relation matrix. If the matching probability of the relation matrix is ​​greater than or equal to the threshold, the target is considered a match; if the matching probability of several relation matrices is less than the threshold, the target is considered a mismatch.

[0039] It should be noted that, using frame images from a standardized dataset as input, a ResNet50 shared feature network is constructed to achieve efficient feature extraction: During network initialization, ResNet50 weights pre-trained on the ImageNet dataset are used, and the first 5 convolutional layers are frozen (this layer mainly extracts general edge and texture features; freezing can reduce overfitting in remote sensing scenes). Only the last 7 convolutional layers are fine-tuned to adapt them to the specific features of remote sensing targets such as agricultural machinery and disaster-stricken buildings (e.g., the metallic texture of agricultural machinery and the reflection features of water bodies). During network operation, feature maps of ResNet50 layers C3 (128×128 resolution, 1024 channels), C4 (64×64 resolution, 2048 channels), and C5 (32×32 resolution, 2048 channels) are extracted. The feature maps of layers C3 and C4 are used by the detection module, and the feature map of layer C5 is used by the association module. By sharing features, redundant feature calculations are reduced by 50%, and finally, a shareable multi-scale spatial feature map is output.

[0040] Using layers C3 and C4 with shared feature maps as input, a dual-attention detection module is constructed to improve the accuracy of remote sensing target localization: For the C3 layer feature map, a network structure of "Conv2d(1024,512,kernel_size=3,padding=1)→ReLU activation function→Conv2d(512,1,kernel_size=1)" is passed sequentially to output a 128×128 center key point heatmap (the higher the pixel value of the heatmap, the greater the probability that the position is the center of the target); For the C4 layer feature map, a network structure of "Conv2d(2048,1024,kernel_size=3,padding=1)→ReLU activation function→Conv2d(1024,4,kernel_size=1)" is passed to output a 128×128×4 corner point heatmap (the 4 channels correspond to the top left, top right, bottom left, and bottom right corner points of the target, respectively). Subsequently, the center coordinates of the target are located by the central heat map, and the target bounding box is generated by combining the coordinates of the four corner points of the corresponding corner heat map. This effectively solves the problem of ambiguity in the localization of irregular remote sensing targets by traditional anchor boxes, and finally outputs an accurate target detection box containing coordinate information.

[0041] Using the C5 layer of the shared feature map and the coordinates of the aforementioned detection boxes as input, a multi-dimensional association module is constructed to achieve cross-frame target matching: features are extracted from both geometric and appearance dimensions, and affinity is calculated. In the geometric dimension, the IoU (threshold range 0.1-0.9, filtering extreme mismatches) and Euclidean distance of the center point are calculated between the current frame detection box and the historical target box (normalized to the [0,1] interval to eliminate scale effects) are calculated. In the appearance dimension, 512-dimensional features of the C5 layer are extracted, and the cosine similarity between the current detection box and the historical target is calculated. According to the weight formula "affinity = 0.4×IoU + 0.2×(1-Euclidean distance) + 0.4×cosine similarity" (the weights are determined by optimization through the validation set), an N×M dimension relationship matrix is ​​generated (N is the number of historical targets, and M is the number of current detection boxes). Finally, a 2-layer GRU (128 hidden layer dimensions) is used to infer the relationship matrix, and an N×M matching probability matrix is ​​output. When the probability is >0.5, it is judged as a match. Finally, the cross-frame target association results with matching / unmatched labels are output.

[0042] The training detection association joint model uses a hierarchical joint loss function combined with backpropagation and gradient descent algorithms to update the model parameters; It should be noted that, based on the detection-association joint model, training parameters were configured to ensure training stability and efficiency: the optimizer AdamW was selected, and the weight decay coefficient was set to 0.001 to suppress overfitting; the initial learning rate was set to 1e-4, and a cosine annealing strategy was adopted (decaying to 0.8 of the previous round every 10 training rounds) to avoid gradient oscillations in the later stages; based on the memory capacity of the NVIDIA A100 80GB GPU, the batch size was set to 8; the total number of training rounds was set to 50, and an early stopping mechanism was introduced—if the multi-object tracking accuracy (MOTA) on the validation set did not improve for 5 consecutive rounds, training was terminated to avoid ineffective training consumption; in addition, to improve the model's generalization ability, data augmentation strategies were adopted for the input frames during the training phase: random horizontal / vertical flipping (probability 0.5), random cropping (cropping ratio 0.7-1.0), and color jittering (brightness ±0.2, contrast ±0.1). The above configuration forms the complete training task parameters.

[0043] The standardized training set, configured training parameters, and the detection-association joint model are used as input for iterative training: each batch of input consists of 8 training set images. After multi-scale features are extracted by the ResNet50 shared feature network, the dual-attention detection module outputs the center key point heatmap, corner heatmap, and target detection box. The multi-dimensional association module outputs the association result based on the detection box and the C5 layer features. The total loss (L_total) is calculated by combining the hierarchical joint loss function. The gradient is automatically calculated by PyTorchautograd, and all trainable parameters of ResNet50, the detection module, and the association module are updated by backpropagation. The model weights are saved every 5 epochs during training. After training, the weights with the highest MOTA on the validation set are selected as the final model parameters to ensure the output of the multi-target tracking model with the best performance.

[0044] The hierarchical joint loss function includes keypoint localization focus loss, bounding box regression loss, and association matching focus loss; the formula for keypoint localization focus loss is as follows:

[0045] in, For the sample size, The heatmap results are obtained based on the true values. This is the heatmap response obtained through keypoint detection. To locate the hyperparameters of the focus loss for key points; The bounding box regression loss formula is expressed as follows:

[0046] in, These are the weight parameters for the bounding box regression loss. These are the true values ​​for the detection box size and offset, respectively. These are the dimensions and offsets obtained by the key point detection module, respectively. The formula for the association matching focus loss is expressed as follows:

[0047] in, For the true labels of the relationship, The model predicts the matching probability. The weight parameters for the association matching focus loss, This refers to the hyperparameters of the association matching focus loss.

[0048] It should be noted that multi-object tracking is decoupled into two tightly coupled sub-tasks: "detection" and "association." Traditional independent optimization methods are prone to task conflicts and feature mismatches. Therefore, we designed a hierarchical joint loss function, which performs end-to-end joint training based on a shared backbone feature network and three highly targeted sub-losses.

[0049] Among these, the Focal Loss technique addresses the core pain point of "high proportion of negative background samples and suppression of positive sample learning by simple negative samples" in remote sensing target center point detection, improving the accuracy of center point localization for small and occluded targets. Focal Loss is chosen instead of ordinary cross-entropy loss—because ordinary cross-entropy loss is dominated by massive amounts of simple negative background samples, causing the model to favor predicting "non-center points," resulting in a soaring false negative rate; Focal Loss introduces a "focusing coefficient" to address this issue. The loss weights are decayed for simple samples (samples whose predicted probabilities are close to the true values), forcing the model to focus on difficult samples (such as the center point of small targets or the center point of occluded target edges). In this embodiment, Take 2, The value of 0.5 was used for validation on the validation set grid search. This value is optimal for the MOTA index in scenarios with small remote sensing targets and occluded targets. It alleviates the problem of negative sample suppression without excessively weakening the constraint effect of background samples, thus avoiding an increase in the model's false detection rate.

[0050] The bounding box regression loss addresses the pain points of "inconsistent magnitudes of size and offset errors, and extreme error interference caused by occlusion / noise" in "regression of corner coordinates of remote sensing targets," thus improving the accuracy of target bounding box generation. L1 loss (absolute value loss) is chosen instead of L2 loss—because remote sensing scenes are prone to interference such as cloud occlusion and sensor noise, which can easily lead to extreme errors in corner regression (e.g., a corner prediction deviation of tens of pixels in a certain frame); L2 loss amplifies the square of extreme errors, leading to gradient explosion or model overfitting outliers, while L1 loss is more robust to outliers and provides smoother gradient updates. Weight parameters are introduced to balance the loss contributions of "detection box size regression error" and "offset regression error"—the magnitudes of size errors (e.g., deviations in the width and height prediction of agricultural machinery) and offset errors (e.g., the offset deviation of corner points relative to the center point) of remote sensing targets may differ (size errors are usually larger than offset errors). The values ​​of the weight parameters are determined through validation set tuning to ensure that the two regression terms are optimized simultaneously, avoiding model bias towards tasks with larger error magnitudes.

[0051] The focus loss algorithm addresses the pain points of "extremely low proportion of positive matching samples and difficulty in matching similar-looking targets" in cross-frame target association, improving the robustness of cross-frame target matching in complex remote sensing scenarios and avoiding ID switching. It uses a binary classification focus loss instead of ordinary binary cross-entropy loss—in association tasks, the number of "unmatched negative samples" (historical targets are not the same as the current detection box) is dozens of times greater than the number of "matched positive samples." Ordinary cross-entropy loss would cause the model to favor predicting "unmatched," resulting in extremely low matching accuracy. The focus loss, through dual constraints of "focus parameters" and "weight parameters," both suppresses simple negative samples and increases the loss weight of positive samples. It is adapted to the "relationship matrix generation → GRU inference" link of the multi-dimensional association module, transforming the association task into a "matched / unmatched" binary classification task. The loss calculation results inversely optimize the feature fusion and inference network of the association module, improving the prediction accuracy of the matching probability matrix.

[0052] S3. Input the remote sensing data to be detected, and use the trained detection association joint model to perform multi-target tracking inference on the remote sensing data to be detected, and obtain the multi-target tracking results.

[0053] The remote sensing test video to be detected is used as input and preprocessed. The preprocessed test video frame sequence and the optimal multi-target tracking model output in step 3 are used as input to perform tracking inference frame by frame: When processing the first frame, the model outputs detection boxes with confidence > 0.3, assigns a unique ID (such as "ID-001" "ID-002") to each detection box and records the initial position; In subsequent frame processing, the detection module first outputs detection boxes with confidence > 0.3 in the current frame, and then the association module reads the position information of historical targets and the appearance features of C5 layer, constructs a relation matrix and uses GRU inference to match the results - the successfully matched targets retain the original ID, the unmatched detection boxes are assigned a new ID as new targets, and if an unmatched historical target does not appear for 3 consecutive frames, it is marked as "disappeared" and removed; At the same time, the position coordinates of each target in the last 10 frames are stored in real time to form a motion trajectory that reflects the dynamic changes of the target. Finally, the ID of the target, the detection box coordinates (x1, y1, x2, y2) and the trajectory sequence of each frame are output.

[0054] Using the frame-by-frame tracking inference results as input, visualization and data output are carried out: using OpenCV tools, detection boxes (different colors are used to distinguish targets with different IDs) and trajectory lines are drawn on the test video frames, supporting real-time playback and frame-level viewing; at the same time, a text file containing target ID, location coordinates, and trajectory sequence is exported to form the final result of multi-target tracking of remote sensing video (including target ID, location, and trajectory).

[0055] Example 2: See Figure 2 As shown, this embodiment is a remote sensing video multi-target tracking system based on detection association joint method, including: The data acquisition and preprocessing module is used to acquire remote sensing data and perform data annotation and preprocessing operations to obtain a standardized sample set. The model building and training module is used to build a detection association joint model, train the detection association joint model based on a standardized sample set, and obtain the trained detection association joint model. The multi-target tracking execution module is used to input the remote sensing data to be detected, and to use the trained detection association joint model to perform multi-target tracking inference on the remote sensing data to be detected, so as to obtain the multi-target tracking results.

[0056] Example 3: To verify the effectiveness of the proposed detection-association joint remote sensing video multi-target tracking method, this embodiment conducts a comparative experiment with existing multi-target tracking methods on the VisDrone2019-MOT validation set.

[0057] The implementation environment configuration for this embodiment is as follows: 1) Hardware environment Processor: Intel Xeon Gold 6348 2.6GHz (or equivalent AMD EPYC processor); Graphics card: NVIDIA A100 80GB (or equivalent GPU, supporting CUDA 11.0 and above). Memory: 128GB DDR4 3200MHz; Storage: 2TB SSD (used to store remote sensing video datasets and model parameters); Input devices: Satellite remote sensing data receiving terminal (such as high-resolution satellite data receiving card) or UAV data transmission module (supporting 4G / 5G real-time transmission).

[0058] (2) Software environment Operating system: Ubuntu 20.04 LTS; Deep learning framework: PyTorch 1.12.0 (or TensorFlow 2.10.0); Dependencies: OpenCV 4.6.0 (video preprocessing), Scikit-learn 1.2.0 (evaluation metric calculation), Matplotlib 3.7.0 (result visualization); Data format: Remote sensing video formats supported include MP4 and AVI (resolution supported: 1080P / 4K, frame rate: 15-30fps).

[0059] First, the accuracy is significantly improved compared to existing methods: On the VisDrone2019-MOT validation set, the multi-target detection mAP@0.5 improves upon existing methods SORT (61.0%) and DeepSORT (66.0%) by 14.0% and 9.0% respectively, with an overall improvement of 8%-15%. For MOTA tracking, it improves upon SORT (58.0%) and DeepSORT (62.0%) by 13.0% and 9.0% respectively, with an overall improvement of 10%-15%. This effectively solves problems such as target occlusion and localization ambiguity in remote sensing scenarios, achieving significant breakthroughs in detection accuracy and tracking robustness. Specific data are shown in Table 1 below. Table 1: Performance comparison of the method on the VisDrone2019-MOT validation set (higher is better, higher FPS is better).

[0060] Secondly, it effectively improves the problem of missed detection of small targets: like Figure 4 As shown, Figure 4 This paper compares the detection results of deepsort and DAUL-MOT on the VisDrone2019-MOT dataset. The figures show that DAUL-MOT demonstrates a significant advantage in small target detection, effectively reducing missed detections of small targets and lowering the false detection rate of similar features. Furthermore, DAUL-MOT maintains accurate target identification and localization even in complex backgrounds, showcasing its robustness and reliability in challenging environments.

[0061] Finally, the algorithm's ability to resist occlusion is effectively improved: like Figure 5-7The figure shows a comparison of the tracking performance of Deepsort and DAUL-MOT on the video sequence uav0000126_00001_v in the VisDrone2019-MOT dataset. The black sedan (target ID-6 in Deepsort and target ID-15 in DAUL-MOT) within the red box in frame 70 of this video sequence was used as the experimental observation object for occlusion resistance performance analysis. As can be seen from the figure, at frame 139, both Deepsort and DAUL-MOT algorithms lost tracking of the black sedan targets ID-6 and ID-15 in frame 70 due to occlusion by trees. By frame 180, the black sedan reappeared, and the target was largely unobstructed; both Deepsort and DAUL-MOT re-identified and resumed tracking. However, the Deepsort algorithm assigned a new target ID, changing it from the initial ID-6 to ID-1576, indicating that it failed to maintain the target's original tracking trajectory. In contrast, DAUL-MOT retained the target's original identification number ID-15 after experiencing occlusion, successfully recovering and continuing its original tracking trajectory, effectively avoiding the problems of target identity switching and trajectory loss caused by occlusion.

[0062] Example 4: This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0063] This computer device can be a server, and its internal structure diagram can be as follows: Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores server data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a remote sensing video multi-target tracking method based on detection correlation joint analysis.

[0064] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] Example 5: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0066] If the functions implemented by the method are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art or the current technical solution, can be embodied in the form of a software product. This current computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0068] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0069] The technical features of this invention not described can be implemented by or using existing technology, and will not be repeated here. Of course, the above description is not a limitation of this invention, and this invention is not limited to the examples above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of this invention should also be within the protection scope of this invention.

Claims

1. A multi-target tracking method for remote sensing video based on detection association, characterized in that, Includes the following steps: Remote sensing data is collected and labeled and preprocessed to obtain a standardized sample set; Construct a detection association joint model, train the detection association joint model based on a standardized sample set, and obtain the trained detection association joint model; Input the remote sensing data to be detected, and use the trained detection association joint model to perform multi-target tracking inference on the remote sensing data to be detected, and obtain the multi-target tracking results.

2. The remote sensing video multi-target tracking method based on detection association joint method according to claim 1, characterized in that, The data annotation includes using a semi-automatic annotation method to annotate the key point coordinates of each target in the remote sensing data, and assigning a unique tracking ID to each target; The key point coordinates include the center point coordinates and corner point coordinates of each target.

3. The remote sensing video multi-target tracking method based on detection association joint method according to claim 2, characterized in that, The preprocessing includes noise reduction, pixel value normalization, size unification, illumination compensation, and timing standardization.

4. The remote sensing video multi-target tracking method based on detection association joint method according to claim 3, characterized in that, The detection association joint model includes a feature extraction module, a dual attention detection module, and a multi-dimensional association module; The feature extraction module uses a ResNet50 network to extract features from a standardized sample set and output a multi-scale spatial feature map. The dual attention detection module is used to generate center point heatmaps and corner point heatmaps based on multi-scale spatial feature maps, determine the center point coordinates and corner point coordinates of the target based on the center point heatmaps and corner point heatmaps, and generate the target bounding box based on the center point coordinates and corner point coordinates of the target. The multi-dimensional association module is used to extract the geometric and appearance features of the target bounding box; A relation matrix is ​​generated based on geometric and appearance features. The relation matrix is ​​then inferred through a GRU network to obtain the relation matrix matching probability. The target matching probability is used to determine whether the target matches, and the target matching result is obtained. The target matching result is then used to associate targets in different frames.

5. The remote sensing video multi-target tracking method based on detection association joint method according to claim 4, characterized in that, The extraction of geometric and appearance features of the target bounding box includes extracting geometric features by calculating the IoU between the target bounding box in the current frame and the target bounding box in historical frames, as well as the Euclidean distance of the center point. Appearance features are extracted by calculating the cosine similarity between the target bounding box in the current frame and the target bounding boxes in historical frames.

6. The remote sensing video multi-target tracking method based on detection association joint method according to claim 5, characterized in that, The step of determining whether a target matches based on the matching probability of the relation matrix includes setting a threshold for the matching probability of the relation matrix. If the matching probability of the relation matrix is ​​greater than or equal to the threshold, the target is determined to match; if the matching probability of several relation matrices is less than the threshold, the target is determined not to match.

7. The remote sensing video multi-target tracking method based on detection association joint method according to claim 6, characterized in that, The training-detection association joint model uses a hierarchical joint loss function combined with backpropagation and gradient descent algorithms to update the model parameters. The hierarchical joint loss function includes keypoint localization focus loss, bounding box regression loss, and association matching focus loss. The formula for the keypoint localization focus loss is as follows: in, For the sample size, The heatmap results are obtained based on the true values. This is the heatmap response obtained through keypoint detection. To locate the hyperparameters of the focus loss for key points; The bounding box regression loss formula is expressed as follows: in, These are the weight parameters for the bounding box regression loss. These are the true values ​​for the detection box size and offset, respectively. These are the dimensions and offsets obtained by the key point detection module, respectively. The formula for the association matching focus loss is expressed as follows: in, For the true labels of the relationship, The model predicts the matching probability. The weight parameters for the association matching focus loss, This refers to the hyperparameters of the association matching focus loss.

8. A remote sensing video multi-target tracking system based on detection association joint method, characterized in that, The steps for implementing the remote sensing video multi-target tracking method based on detection association joint as described in any one of claims 1 to 7 when executing the computer program include: The data acquisition and preprocessing module is used to acquire remote sensing data and perform data annotation and preprocessing operations to obtain a standardized sample set. The model building and training module is used to build a detection association joint model, train the detection association joint model based on a standardized sample set, and obtain the trained detection association joint model. The multi-target tracking execution module is used to input the remote sensing data to be detected, and to use the trained detection association joint model to perform multi-target tracking inference on the remote sensing data to be detected, so as to obtain the multi-target tracking results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the remote sensing video multi-target tracking method based on detection association joint as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the remote sensing video multi-target tracking method based on detection association joint as described in any one of claims 1 to 7.

Images