A Deep Learning-Based Multi-Target Tracking Optimization Algorithm Based on RK3588

Abstract

This invention relates to the intersection of computer vision and embedded tracking technology, and optimizes and improves the RK3588 embedded platform to address issues such as poor scene adaptability and insufficient robustness in multi-target tracking. A lightweight YOLOv5-RK detector is employed to reduce computational overhead, a forward differential adaptive thresholding mechanism is designed to adapt to complex scenes, Kalman filtering is optimized through dynamic estimation of Q / R parameters to improve motion prediction robustness, and IOU is replaced with DIoU to optimize data association matching. The improved ByteTrack-RK algorithm achieves a 2.91% improvement in MOTA, a 5.57% improvement in IDF1, and a 111 reduction in IDs on the MOT dataset. The resulting deep learning-based multi-target tracking RK3588 embedded platform achieves a speed of 45 FPS when tracking 720P video streams, meeting the real-time multi-target tracking requirements of embedded applications such as UAV aerial surveillance and intelligent transportation.

Description

Technical Field

[0001] This invention relates to the intersection of computer vision and embedded systems technology, specifically a deep learning-based multi-target tracking optimization solution adapted to the Rockchip RK3588 embedded platform. Through hardware-software co-design, it achieves high-precision, low-latency real-time tracking. This method can be widely applied to scenarios requiring real-time, high-precision, and robust continuous multi-target tracking in computationally limited environments, such as drone aerial surveillance, intelligent urban transportation, and security for high-density crowds. Background Technology

[0002] With the widespread adoption of applications such as drone aerial photography and intelligent surveillance, the demand for multi-target tracking (MOT) technology on embedded platforms is becoming increasingly urgent. However, existing mainstream algorithms face a seemingly irreconcilable "triple contradiction" when deployed on low-power embedded chips such as the RK3588: First, there is a contradiction between fixed thresholds and dynamic scenes: Traditional algorithms (such as ByteTrack and SORT) use fixed confidence thresholds to divide detection boxes, which cannot cope with the problems that are common in aerial photography scenes, such as drastic changes in lighting, large differences in target scale, motion blur and mutual occlusion. This leads to drastic fluctuations in detection confidence, which can easily result in missed detections or false detections (corresponding to the adaptive threshold module of this invention).

[0003] Second, there is a contradiction between fixed filtering parameters and target maneuverability: As the core motion prediction model, the Kalman filter's process noise covariance (Q) and observation noise covariance (R) usually rely on empirical presets. When the target maneuvers rapidly or the scene is dynamically complex, the fixed Q / R parameters cannot accurately reflect the true noise level, leading to predicted trajectory drift and thus triggering frequent identity switching (IDs) (corresponding to the adaptive Kalman filter module of this invention).

[0004] Third, the data association accuracy is low: Algorithms such as DeepSORT improve association accuracy by introducing a ReID model, but its huge computational overhead causes the inference speed on the RK3588 platform to plummet to below 10 FPS, which cannot meet the real-time requirements. While the lightweight ByteTrack can guarantee speed, the detector it uses is not optimized for embedded hardware, and its fixed threshold, IoU matching, and empirical Kalman parameters perform poorly in complex scenarios, resulting in insufficient overall tracking accuracy and stability (corresponding to the DloU matching and lightweight co-design of this invention).

[0005] However, when existing multi-target tracking technologies are deployed on embedded platforms such as the RK3588, they suffer from model redundancy, high data transfer overhead, and rigid association strategies, making it difficult to balance accuracy and speed. Especially on embedded platforms like the RK3588, which combines multi-core NPUs and CPUs with heterogeneous computing units, existing solutions either fail to achieve real-time performance due to computational complexity or suffer from a loss of accuracy and robustness due to coarse algorithms, making it difficult to achieve a balance among these three aspects. Summary of the Invention

[0006] This invention recognizes that on heterogeneous embedded platforms like the RK3588, the aforementioned contradictions are intertwined, and improvements in any single aspect are unlikely to overcome the overall performance bottleneck. Therefore, end-to-end collaborative design, from the algorithm core to hardware scheduling, is necessary to ensure that the optimization of the detection, prediction, and correlation stages mutually support each other and form a closed loop, in order to achieve a balance between accuracy, robustness, and real-time performance.

[0007] The present invention transforms the above-mentioned "triple contradiction" into three specific synergistic optimization objectives: (1) synergy between detection efficiency and accuracy; (2) synergy between motion prediction universality and adaptability; and (3) synergy between association accuracy and computational overhead.

[0008] The specific technical solution of the present invention includes the following steps: Step 1: Build and deploy a lightweight, highly adaptable YOLOv5-RK detector To improve inference efficiency on the RK3588 embedded platform and enhance the detection capability of small-scale targets in aerial photography scenes, this invention uses YOLOv5s as the base model and constructs a dedicated detector, YOLOv5-RK, through model structure reconstruction and lightweight design. Specific optimizations include the following three aspects: (1) Lightweight Reconstruction of the Backbone Network: The Ghost Bottleneck module replaces some standard convolutional layers in the backbone network, utilizing its linear operation mechanism to generate feature maps, significantly reducing computational overhead while maintaining feature representation capabilities. For example... Figure 3 As shown, the Ghost Bottleneck module reconstructs the C3_1 structure in YOLOv5: with a stride of 1, this module consists of two Ghost modules concatenated and combined with residual connections to maintain feature representation capability; with a stride of 2, depthwise separable convolutions are introduced for downsampling to balance feature extraction and computational efficiency. Furthermore, as... Figure 4 As shown, the C3_Ghost module built on GhostBottleneck retains rich feature representations while reducing redundant computation through a more efficient feature generation mechanism, effectively avoiding the significant performance degradation caused by the lightweighting of the backbone network.

[0009] (2) Model channel pruning: Based on the lightweight structure, a channel pruning strategy is further implemented to automatically identify and remove redundant channels that contribute little to the final detection performance, thereby compressing the model size and reducing memory usage and inference latency.

[0010] (3) Enhanced small target detection capability: Based on the original Feature Pyramid Network (FPN), a Tiny Pyramid Network (TPN) structure specifically designed for small targets is introduced, and a shallow detection head with a resolution of 160×160 is added. This design enhances the feature extraction and localization capability of small targets in images, effectively alleviating the problem of missed detection caused by the variable target scale in aerial photography scenarios.

[0011] After the aforementioned collaborative optimization, the number of parameters in the resulting YOLOv5-RK model was reduced to 12.2M, and the floating-point operation (FLOPs) was reduced to 14.8G. On the VisDrone2019 dataset, the model achieved an AP50 accuracy of 37.6%; when deployed on the RK3588 platform, it can achieve real-time inference of up to 60 FPS for 720P video streams, providing efficient and reliable detection input for subsequent multi-object tracking modules.

[0012] Step 2: Design a forward differential adaptive threshold mechanism To address the issue that traditional multi-object tracking algorithms (such as ByteTrack) using fixed confidence thresholds (typically 0.3–0.5) struggle to balance recall and false positive suppression in complex scenarios, this invention proposes an adaptive dynamic threshold setting method based on forward differencing. The process is as follows: Figure 3 As shown, the specific steps include: (1) Obtain all detection boxes and their confidence scores for a single frame output by the detector; (2) Arrange the confidence levels in descending order to form a discrete sequence. ; (3) Calculate the forward difference ; (4) Select the one that makes Index to get the maximum value and the corresponding confidence level Set the dynamic threshold T for the current frame, and calculate the threshold independently for each frame, without the need for manual preset.

[0013] The core of this method lies in the fact that there is usually a significant gradient difference in the confidence distribution between real targets and low-quality noise. High-quality targets are concentrated in the high-confidence region, and then the confidence rapidly decreases into the noise region. Forward differencing can accurately capture this "sharp drop point" (i.e., (maximum point), such as Figure 4As shown by the red dot. Setting the threshold at this critical position can effectively distinguish between high-confidence true targets and low-confidence interference items, significantly suppressing false detections while preserving recall capability.

[0014] Since this mechanism is driven entirely by real-time detection output, it can adaptively respond to changes in target density, background complexity, and imaging conditions, significantly improving the robustness and generalization ability of the algorithm in diverse real-world scenarios.

[0015] Step 3: Optimize Kalman filter to improve motion prediction robustness Traditional Kalman filters in multi-target tracking typically employ pre-defined fixed process noise covariance matrix Q and observation noise covariance matrix R, which struggles to adapt to dynamic target changes and complex observation environments, easily leading to trajectory drift and identity switching. To overcome this deficiency, this invention proposes an online adaptive estimation mechanism based on innovation and residuals, dynamically updating the Q and R parameters, such as... Figure 5 As shown, the specific implementation steps are as follows: (1) Initialization phase: perform state initialization and state error initialization; (2) In the prediction stage, the predicted value at time k-1 is used to predict the value at time k based on the state transition matrix, and the prior covariance matrix at time k is calculated. (1) (2) (3) In the formula, For residuals, For residual covariance, Let k be the observation value at time k. Let H be the best estimate at time k, and H be the observation matrix.

[0016] Equation (3) shows the estimation of the R value. To estimate R, the residuals... As shown in formula (1), the residual is calculated from the observed value and the state estimate at time k, and is the difference between the actual observed value and the predicted state estimate. Through the residual, we can understand the difference between the accuracy of the prediction and the actual observation, and thus infer the magnitude of the observation noise R.

[0017] (3) In the update phase, the innovation and the estimated value at time k are calculated based on the predicted value at time k, and the residual is calculated using the estimated value at time k. The estimated values ​​of Q and R are then calculated using the calculated innovation and residual. (4) (5) In the formula, For new information.

[0018] Equation (5) shows an estimate of the Q value. (News) As shown in Equation (4), innovation measures the quality of a prediction by calculating the difference between the predicted and actual observed values. Unlike residuals, innovation is based on prediction rather than estimation, thus reflecting the error in the prediction phase and thereby estimating the magnitude of the system noise Q.

[0019] (6) To make it smoother, a low-pass filter is used to obtain the final estimates of Q and R, as shown in Equation (6), where in Equation (6) This is used to balance the weights of historical estimates and current observations. Experiments have verified that... When the value of is between 0.1 and 0.5, the system can achieve a good balance between tracking stability and adaptability to dynamic changes. When the value is below 0.1, the filtering process is too sensitive to noise; when... When the value is higher than 0.5, the parameter updates are too slow to adapt to the target's rapid maneuvers. The optimal value for overall performance in this application scenario is... =0.3.

[0020] Step 4: Optimize data association and matching to improve detection performance in occluded scenes. This invention optimizes the data association in the ByteTrack algorithm to improve its stability and robustness. IoU matching plays a crucial role in the entire data association process of the ByteTrack algorithm. IoU matching is used to calculate the overlap between two bounding boxes. It is typically used to measure the degree of matching between the model-predicted bounding box and the ground truth bounding box. The calculation formula is shown in formula (7).

[0021] (7) IoU (Intersection over Union) can intuitively reflect the degree of matching between the detected target and the actual situation. However, IoU matching only calculates the area ratio of the overlapping region and cannot accurately reflect the spatial relationship between the center points of the bounding box. When the target is occluded or the targets are very close together, the IoU may incorrectly miss the target due to insufficient overlapping area. Therefore, this section improves IoU matching.

[0022] (8) The calculation formula for DIoU (Distance Intersection over Union) is shown in Equation (8). It is an improvement on the traditional IoU, adding regression of the Euclidean distance between the center points of the target boxes to accelerate the convergence process. The penalty term of DIoU is based on the ratio of the distance between the center points of the boxes to the diagonal distance, effectively avoiding the closure problem caused by the target boxes being far apart when the IoU is large. Compared with IoU, DIoU can capture the spatial relationship between targets more accurately, especially when the overlapping area is insufficient or the similarity between targets is low, avoiding mismatch and target loss. Therefore, in the ByteTrack algorithm, using DIoU as the matching criterion not only improves the accuracy of target matching, but also enhances the robustness and stability of the algorithm in complex scenes.

[0023] To address the issue of IoU matching failure in densely occluded scenes, this invention replaces the IoU matching criterion in ByteTrack with DIoU. The steps are as follows: (1) In the data association stage, the original matching criterion based on intersection-over-union ratio (IoU) in the ByteTrack algorithm is replaced with a matching criterion based on distance intersection-over-union ratio (DIoU); (2) For each pair of trajectory prediction boxes and detection boxes to be matched, calculate its DIoU value. The formula for calculating DIoU is: .in, and Here, ρ represents the center points of the predicted bounding box and the observed bounding box, respectively, and ρ is the Euclidean distance. The length of the diagonal of the smallest bounding rectangle containing the two boxes; (3) Construct a cost matrix using the calculated DIoU value and use it as input to the Hungarian algorithm or linear assignment algorithm to complete the first matching between the high-confidence detection box and the existing trajectory; (4) For trajectories that are not successfully associated after the first match, continue to use the DIoU metric to perform a second match with the low confidence detection box set in order to recover the target that was temporarily lost due to the confidence threshold filtering. (5) By introducing a normalized penalty term for the center point distance, DIoU can still effectively distinguish bounding boxes that are spatially close but not the same target when the overlapping area between targets is small or partial occlusion occurs, thereby reducing the false match rate and reducing target loss.

[0024] Step 5: Integrate and optimize modules to form the ByteTrack-RK algorithm and deploy it. Using the original ByteTrack as the baseline model, four major modules—lightweight detector (YOLOv5-RK), adaptive threshold, adaptive Kalman filter, and DIoU matching—are integrated to form the final multi-target tracking optimization algorithm ByteTrack-RK. This achieves full-process optimization from detection and prediction to matching, and adapts to the computing power and real-time requirements of the RK3588 embedded platform.

[0025] To address the need for deploying algorithms on the RK3588 platform, we completed the entire conversion process from trained models to embedded inference models using the RKNN-Toolkit2 toolchain provided by Rockchip. Since the RKNN environment does not directly support PyTorch models, the PyTorch models needed to be converted to ONNX format first, and then further converted to RKNN models. The specific steps are as follows: (1) Perform the conversion from PyTorch to ONNX: initialize the PyTorch environment and load the trained model weights; define the input and output tensor shapes and data types of the model and clarify the inference interface; call the torch.onnx.export interface to export the PyTorch model as an ONNX format model; (2) Perform ONNX to RKNN conversion: Load the ONNX model in the environment with RKNN-Toolkit2 configured; configure the RKNN construction parameters according to the model structure, including input / output node names, sizes, etc.; generate and export the RKNN model file adapted to RK3588; (3) After the model conversion is completed, the multi-threaded asynchronous inference technology is combined to achieve efficient deployment on the RK3588 platform, giving full play to the collaborative computing capabilities of NPU and CPU and improving the overall inference efficiency.

[0026] Beneficial effects Through the aforementioned collaborative optimization, this invention achieves significant technical advancements on the RK3588 embedded platform, specifically in a comprehensive improvement in accuracy, robustness, and real-time performance. The contribution metrics of each optimization module are analyzed below: (1) Significantly improved accuracy: On the MOT17 dataset, the ByteTrack-RK algorithm, which integrates all optimization modules, achieves a multi-target tracking accuracy (MOTA) of 81.2%, representing an absolute improvement of 2.91% and a relative improvement of 3.69% compared to the original ByteTrack baseline (78.9%). Its identity recognition F1 score (IDF1) reaches 79.5%, representing an absolute improvement of 4.2% and a relative improvement of 5.57% compared to the baseline (75.3%).

[0027] Decomposition of contributions from each module (based on ablation experiments): The lightweight YOLOv5-RK detector is the core of the accuracy improvement. Its more efficient feature extraction and small target enhancement design alone contributed approximately 2.41% to the MOTA improvement and 4.24% to the IDF1 improvement. The forward differential adaptive thresholding mechanism reduces false matches by dynamically filtering effective detection boxes, contributing approximately 0.88% to the MOTA improvement and reducing 21 identity switching (IDs). DloU-optimized data association matching utilizes center point distance information to improve box matching accuracy in occluded and dense scenes, contributing approximately 1.14% to the MOTA improvement and 1.06% to the IDF1 improvement.

[0028] The data above shows that each module has played a clear and quantifiable positive role in improving accuracy indicators in different dimensions.

[0029] (2) Enhanced tracking robustness: The adaptive Kalman filter module enables the algorithm to dynamically adapt to changes in target motion and the observation environment by estimating and smoothly updating the noise covariance Q / R online. Although the performance fluctuates slightly when compared with known optimal static parameters, its core value lies in eliminating dependence on empirical parameters, greatly improving the algorithm's adaptability and stability in real-world, unknown scenarios.

[0030] After integrating all optimizations, the number of identity switching (IDs) on the MOT17 dataset was reduced from 2875 to 2764, a significant reduction of 111, demonstrating the enhancing effect of module collaboration on trajectory coherence.

[0031] (3) Breakthrough in real-time performance: Through an algorithm-hardware co-optimization system, efficient end-to-end inference was achieved on the RK3588 platform. Lightweight detection and hybrid quantization: The YOLOv5-RK detector itself can achieve a detection speed of 60 FPS for 720P video streams, providing ample time for the tracking process. Multi-core NPU asynchronous pipeline: Through the specialized division of labor among three NPU cores (detection, correlation update, and state management) and thread pool scheduling, detection and tracking tasks are deeply parallelized.

[0032] Ultimately, the complete system achieves a tracking speed of up to 45 FPS for 720P video streams and 35 FPS for 1080P video streams, fully meeting the real-time requirements of scenarios such as drones and intelligent transportation, and achieving a speed improvement of more than 3 times compared to the unoptimized baseline.

[0033] (4) Resource consumption has been significantly reduced: The optimized algorithm has a peak memory footprint of only 26.8 MB on the RK3588 platform, which is more efficient than the baseline solution. This demonstrates the effectiveness of the lightweight design and resource scheduling optimization, and is conducive to long-term stable operation in resource-constrained embedded platforms.

[0034] (5) Module synergy effect: The core advancement of this invention stems from the deep collaborative closed loop formed by the modules within the resource constraints of the RK3588 platform: the lightweight detector (YOLOv5-RK) provides high-speed, high-quality initial detection, laying the foundation for the entire system; adaptive threshold dynamic purification of detection results directly improves the input quality of subsequent DloU matching, reducing erroneously associated candidate pairs; the more accurate association results provided by DloU matching enable adaptive Kalman filtering to estimate noise parameters based on more reliable observations, thereby generating more accurate state predictions; finally, a multi-core NPU pipeline strategy based on functional partitioning (see claim 9) efficiently maps the above algorithm modules onto the hardware, enabling the entire closed loop to run in real time. This tightly coupled design of "detection → filtering → matching → prediction → hardware scheduling" is key to solving the problem of "at the expense of one aspect" in embedded multi-target tracking. This deep collaborative solution across the algorithm and hardware layers for a specific embedded platform (RK3588) transcends the general approach of simply improving existing algorithms or independently optimizing hardware scheduling, demonstrating significant progress. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the YOLOv5-RK model of the present invention; Figure 2 Target detection results in different scenarios in the embodiments of the present invention; Figure 3 This is a flowchart of the adaptive threshold dynamic calculation process of the present invention; Figure 4 This is an example diagram showing the adaptive threshold calculation results of the present invention; Figure 5 This is the adaptive estimation process for noise and measurement covariance in this invention; Figure 6 Visualization of the target tracking results of the algorithm in the MOT17-04 video sequence in this embodiment of the invention; Figure 7 Visualization of the target tracking results of the algorithm in the MOT20-02 video sequence in this embodiment of the invention; Figure 8 The algorithm in this embodiment of the invention is shown in the target tracking result diagram of aerial video stream on the RK3588 platform. Detailed Implementation

[0036] The present invention will now be further described in conjunction with the embodiments and accompanying drawings: The specific implementation is as follows: Step 1: Data Preparation and Environment Setup The MOT17 and MOT20 datasets provided by the MOTchallenge official website were used, containing 14 video sequences (7 training sets and 7 test sets), totaling 11235 frames. Annotation information included target bounding boxes, trajectory IDs, and categories. The training set was divided in an 8:2 ratio, with 5 sequences used for training and 2 for validation. The test set was used for final performance evaluation. Images were uniformly scaled to 640×640 resolution, and the annotation format was converted to YOLO format. The experimental environment included: a training workstation (dual RTX2080Ti GPUs), a development PC (Ubuntu virtual machine), and a validation platform (RK3588 development board, ITX-3588J motherboard, 16GB RAM).

[0037] Step Two: Module Implementation and Integration Lightweight detector deployment: Load the trained YOLOv5-RK model and convert it to the RK3588 compatible RKNN format using RKNN-Toolkit2 to ensure that the detection speed can reach 60 FPS under 720P video stream; Design an adaptive threshold module: Write a forward difference calculation function, take the detection box confidence list as input, and output a dynamic threshold; Adaptive Kalman filter implementation: Add dynamic update functions for Q and R to the original Kalman filter algorithm. After each state update, calculate and update Q and R based on the residual and the new information. DIoU Matching Replacement: Rewrite the data association function of the ByteTrack baseline network, replace IOU calculation with DIoU calculation, and retain ByteTrack's secondary matching logic.

[0038] Step 3: Algorithm Integration and Deployment The four core modules optimized in this invention—the YOLOv5-RK detector, adaptive threshold partitioning, adaptive Kalman filter prediction, and DIoU quadratic matching—are integrated to construct a complete improved multi-target tracking algorithm. This integrated algorithm is further named ByteTrack-RK, and an executable file suitable for the RK3588 embedded platform is generated through cross-compilation. To fully leverage the platform's collaborative computing capabilities, an asynchronous inference mechanism with a thread pool is designed, configuring a 6-thread NPU and CPU for parallel processing to achieve efficient resource utilization.

[0039] Practical tests were conducted on aerial video streams at both 720P and 1080P resolutions on the RK3588 platform. Experimental results show that the proposed algorithm significantly improves inference speed while maintaining high tracking accuracy, demonstrating excellent real-time processing capabilities. This fully validates the effectiveness and engineering applicability of the proposed method in real-world embedded scenarios.

[0040] To verify the effectiveness of the multi-target tracking optimization algorithm adapted to the RK3588 embedded platform provided by this invention, simulation experiments were also conducted.

[0041] Step 4: Implementation of RK3588 Platform-Specific Optimizations and Deployment To achieve efficient deployment and real-time inference of the ByteTrack-RK algorithm on the RK3588 embedded platform, this implementation method has undergone in-depth adaptation and optimization in terms of model conversion, hardware scheduling and system integration.

[0042] Model Conversion and Mixed-Precision Quantization Strategy: Deploying the trained lightweight detector (YOLOv5-RK) to the RK3588 platform requires using the RKNN-Toolkit2 toolchain provided by Rockchip to convert the training framework into an embedded inference model. First, the PyTorch model weights are converted to the open ONNX intermediate format. Then, the ONNX model is loaded into the RKNN-Toolkit2 environment for compilation and optimization tailored to the NPU hardware characteristics.

[0043] To maximize NPU computational efficiency while ensuring detection accuracy, this implementation employs a hybrid precision quantization strategy. Specifically, INT8 low-bit quantization is used for the computationally intensive and relatively robust backbone network convolutional layers to significantly reduce computational latency and memory usage. However, for modules that directly impact small target detection performance, such as the newly added 160×160 scale shallow detection head and the upsampling and concat layers in the FPN / PAN structure responsible for multi-scale feature fusion, which are extremely sensitive to numerical precision, FP16 floating-point precision is retained to preserve rich detail information and gradient characteristics. The quantization calibration process is performed using 100 representative images from the training dataset to ensure that the quantization parameters accurately reflect the real data distribution.

[0044] Multi-core NPU asynchronous inference and thread pool scheduling: The RK3588 chip integrates three independent NPU computing cores (NPU_0, NPU_1, NPU_2). To fully leverage its parallel computing potential, this implementation design adopts an asynchronous inference scheduling mechanism based on a thread pool.

[0045] Model Instantiation and Core Binding: During system initialization, the optimized RKNN model file is loaded three times, creating three independent inference instances (rknn_context). Each instance is explicitly bound to a different physical NPU core via the RKNN API, achieving hardware-level isolation and parallelization of computational load.

[0046] Pipeline task scheduling: A producer-consumer task queue is constructed. The video decoding module acts as the producer, adding sequentially acquired video frames into the queue. A thread pool consisting of multiple worker threads acts as the consumer, retrieving frames from the queue in a round-robin manner and distributing them to their bound NPU instances for asynchronous inference. This architecture ensures continuous and parallel processing of the video stream, effectively hiding single-inference latency and achieving extremely high hardware utilization and throughput.

[0047] Efficient resource utilization: This scheduling mechanism avoids the situation where a single NPU core is idle while other cores are overloaded, enabling the three NPU cores to work in parallel with a balanced load. This is a key technical guarantee for achieving real-time high frame rate processing (such as 720P@45FPS).

[0048] System Integration and Performance Verification: The optimized RKNN detection engine was integrated with the adaptive threshold module, adaptive Kalman filter module, and DIoU data association module implemented in C++ and executed on the CPU. The resulting system was compiled and built into a complete target tracking application that can run directly on the RK3588 platform (running Ubuntu).

[0049] In the final performance verification, this implementation method was tested end-to-end on the RK3588 development board using aerial video streams at 720P and 1080P resolutions. As shown in Table 5 (the performance comparison table in the original specification), the optimized ByteTrack-RK algorithm maintained high tracking accuracy (MOTA 81.2%) while significantly improving inference speed, achieving 45 FPS for 720P video streams and 35 FPS for 1080P video streams, with a peak runtime memory usage of only 26.8MB. These data fully validate the effectiveness of the entire optimization scheme, from model lightweighting and hybrid quantization to multi-core asynchronous scheduling, enabling the algorithm to fully meet the stringent requirements of embedded platforms for real-time performance, accuracy, and low resource consumption.

[0050] It is particularly important to note that the collaborative performance of the aforementioned algorithm modules is ultimately achieved through the multi-core NPU pipeline strategy based on functional partitioning, as defined in claim 9. By embedding detection, correlation / filtering updates, and state management tasks into different NPU cores, data transfer and pipeline downtime overhead are significantly reduced, transforming the collaborative design at the algorithm level into actual end-to-end performance improvements. This algorithm-hardware collaborative optimization is something that those skilled in the art would find difficult to conceive of simply improving the algorithm or optimizing hardware scheduling.

[0051] This pipeline strategy is not only a parallel computing method, but also a hardware enabler for the aforementioned algorithmic collaboration. It arranges modules with data dependencies (such as those that can only be associated after detection) to be executed in parallel on different cores, and overlaps computation and data transfer through an efficient queuing mechanism, thereby ensuring that the collaborative design at the algorithm level will not fail due to hardware scheduling bottlenecks.

[0052] As a further optimization of the aforementioned multi-core asynchronous inference scheduling mechanism, a preferred embodiment may also employ a pipeline parallel strategy based on functional partitioning. Specifically, the three NPU cores are specialized: the first NPU core (NPU_0) is dedicated to executing the forward inference of the YOLOv5-RK detector; the second NPU core (NPU_1) is configured to perform parallel data association calculations (including DIoU matching) and Kalman filtering state updates; and the third NPU core (NPU_2) is responsible for caching and managing the state queue of historical trajectories. By optimizing task allocation and data flow, detection results can be efficiently transmitted to subsequent processing units, aiming to further reduce pipeline pauses and achieve lower latency end-to-end tracking. Those skilled in the art will understand that the allocation of the above core functions can be flexibly adjusted according to actual hardware characteristics and task load.

[0053] Experimental environment setup The specific experimental environment is shown in Table 1. The workstation is used for algorithm training, the PC virtual machine is used for software development and cross-compilation, and finally the algorithm is verified on the RK3588 platform.

[0054] Table 1 Experimental Environment Experimental content This experiment verifies the effectiveness of each optimization module through ablation experiments and verifies the superiority of the overall solution through comparative experiments. The test scenarios include various typical aerial photography scenarios on the MOT17 and MOT20 datasets.

[0055] Evaluation indicators To evaluate the performance of multi-object tracking algorithms, this invention uses the CLEAR metric, which includes Multiple Object Tracking Accuracy (MOTA), Identification F1 (IDF1), Higher Order Tracking Accuracy (HOTA), Mostly Tracked (MT), Mostly Lost (ML), and Identity Switch (IDs). MOTA, IDF1, and IDs are three important comprehensive evaluation metrics. MOTA focuses on tracking accuracy, while IDF1 and IDs focus on correlation performance.

[0056] (1) IDF1 refers to the accuracy of ID information during target tracking. The calculation formula is shown in formula (9).

[0057] (9) In the formula, IDTP represents the number of true positives of the target identifier, IDFP represents the number of false positives of the target identifier, and IDFN represents the number of false negatives of the target identifier.

[0058] (2) IDs refers to the number of times the target's ID changes during the tracking process.

[0059] (3) MOTA refers to the accuracy of multi-target tracking, which intuitively shows the accuracy of the successful matching between the real target and the predicted target. Its calculation formula is shown in formula (10).

[0060] (10) In the formula, FN (False Negatives) represents the number of real targets that were not successfully matched. FP (False Positives) represents the number of tracked targets that were incorrectly predicted. GT (Ground Truth) represents the total number of labeled targets.

[0061] Simulation test results This invention uses the ByteTrack algorithm as the benchmark algorithm for multi-target tracking and the YOLOv5s algorithm as the benchmark detector algorithm. Ablation experiments were conducted on the MOT17 dataset to verify the effectiveness of the proposed method combining lightweight detectors, adaptive thresholds, adaptive Kalman filtering, and optimized data association matching. Table 2 shows the ablation experiment results of the improved method proposed in this invention.

[0062] Table 2 Ablation Experiment Results As shown in the table, after switching to a lightweight detector, the MOTA (Motion Over-The-Air) index of the multi-target tracking algorithm improved by 2.41%, the IDF1 index improved by 4.24%, and the IDs index remained basically unchanged. This indicates that the improved YOLOv5s-RK detector can not only effectively improve the accuracy of multi-target tracking, but also significantly reduce the number of target identity switching, thereby enhancing tracking stability. After introducing an adaptive confidence threshold, the MOTA index improved by 0.88%, the IDF1 index remained basically unchanged, and the IDs index decreased by 21. This shows that the adaptive threshold adjustment strategy, by dynamically estimating the confidence of the detection box, helps to improve the accuracy of data association, making the matching of trajectory and detection results more flexible and reliable.

[0063] Experimental results show that the improved Kalman filter slightly reduces the MOTA and IDF1 scores of the multi-target tracking algorithm, while slightly increasing the IDs score. This is mainly because the Q and R parameters in the Kalman filter were set based on optimal values ​​derived from experience and experiments during testing. The core idea of ​​adaptive Kalman filtering is to dynamically estimate and adjust the Q and R parameters according to the actual situation. Knowing the optimal parameters, adaptive Kalman filtering cannot further improve tracking accuracy, leading to a slight decrease in performance. However, in practical applications, the optimal values ​​of Q and R are usually unknowable in advance. In this case, adaptive Kalman filtering can dynamically adjust the parameters according to the target's motion characteristics, thereby significantly improving the algorithm's robustness and adaptability. The experimental results show that using dynamically estimated Q and R parameters results in a smaller decrease in algorithm accuracy compared to using optimal values. This demonstrates the accuracy of the dynamic Q and R parameter estimation method of this invention and verifies the feasibility and potential advantages of adaptive Kalman filtering.

[0064] After optimizing the correlation matching algorithm, experiments show that the MOTA (Motion Over-The-Air) index of the multi-target tracking algorithm improved by 1.14%, the IDF1 index improved by 1.06%, while the IDs index remained basically unchanged. This indicates that replacing the IoU (Intersection over Union) matching strategy with the DIoU (Division over Union) matching strategy can utilize center point distance information for more accurate matching when the overlapping area of ​​the target boxes is insufficient, thereby further improving the accuracy of multi-target tracking. After comprehensively applying all improvements, the MOTA of the multi-target tracking algorithm improved by 2.91%, the IDF1 index improved by 5.57%, and the IDs index decreased by 111. This shows that the improved method proposed in this invention can effectively improve the tracking accuracy and stability of the ByteTrack algorithm, which is beneficial for deployment on embedded platforms and realizing efficient and accurate multi-target tracking tasks.

[0065] This invention compares the improved ByteTrack-RK algorithm with several excellent multi-target tracking algorithms in the industry. The comparative experiments were verified on two classic datasets, MOT17 and MOT20.

[0066] Table 3 shows the experimental results of different algorithms on the MOT17 test set. ByteTrack-RK is the algorithm presented in this chapter. According to the data in the table, the algorithm of this invention demonstrates significant advantages in multi-target tracking tasks. In terms of the MOTA metric, the algorithm of this invention achieves 81.2, a significant improvement compared to the original ByteTrack algorithm's 78.9, DeepSORT's 78.6, and other comparative algorithms. This indicates that the algorithm of this invention can better track multiple targets and effectively reduce missed detections and false detections, reflecting improved tracking accuracy. In terms of the IDF1 metric, the algorithm of this invention achieves the highest value of 79.5, better than the original ByteTrack algorithm's 76.8 and other comparative algorithms, indicating that the algorithm of this invention can more accurately match the identity of the target, improving the stability and accuracy of tracking. In terms of IDs, the algorithm proposed in this invention performs excellently, with a value of 2764, better than most algorithms and only slightly higher than the DeepSORT algorithm. Analysis shows that DeepSORT introduces a ReID network to extract the appearance features of the target, thereby reducing ID jump phenomena. However, this method also increases the computational load, leading to certain limitations in performance when applied to embedded platforms. Therefore, although the algorithm proposed in this invention is slightly inferior to DeepSORT in terms of the number of identity switching, it has the advantage of being applied on embedded platforms due to its optimized computational efficiency.

[0067] Table 3 Experimental results of different algorithms on the MOT17 test set Table 4 shows the experimental results of different algorithms on the MOT20 test set. The data shows that the algorithm proposed in this invention still maintains a significant advantage in both MOTA and IDF1. The MOT20 dataset is more complex than MOT17, containing higher density pedestrians, complex occlusion situations, and more challenging scenes. In this more complex scenario, the ByteTrack-RK algorithm, with its optimized tracking strategy and efficient computation, still maintains excellent MOTA and IDF1 performance, demonstrating its robustness and stability across different dataset types. Although the IDs index of this invention's algorithm (2875) is slightly higher than DeepSORT's (2645) on the MOT20 dataset, this does not imply a major problem with identity switching. In fact, DeepSORT effectively reduces ID switching by introducing a ReID network to extract the target's appearance features, but this method increases computational cost, leading to performance bottlenecks in practical applications on embedded platforms. Therefore, although the identity switching value of the ByteTrack-RK algorithm proposed in this invention is slightly higher on MOT20, its efficiency and real-time performance advantages on embedded platforms are still obvious. It can maintain high tracking accuracy and stability while ensuring low computational overhead.

[0068] Table 4. Experimental results of different algorithms on the MOT20 test set. Visualization Analysis To more clearly demonstrate the tracking performance of the improved algorithm of this invention, the results of a deep learning multi-target tracking optimization algorithm based on RK3588 are presented and analyzed. Target tracking results from video sequences 04 of the MOT17 dataset and 02 of the MOT20 dataset are used as examples.

[0069] like Figure 6 The results show the target tracking performance of the algorithm on the MOT17-04 video sequence. First, in occlusion scenarios, as shown in Figure (a), the target with Id 1 can still be accurately tracked even after being occluded by a utility pole. Similarly, as shown in Figure (c), the algorithm can continue to track the target's state even when the target with Id 44 is occluded by the target with Id 42, unaffected by the occlusion. Furthermore, as shown in Figure (d), the target with Id 3 can still maintain stable tracking even when occluded by a utility pole. These results demonstrate that the algorithm has advantages in dealing with occlusion situations, effectively handling interference caused by target occlusion and maintaining high tracking accuracy.

[0070] Secondly, regarding the ID jump problem, experimental results show that the algorithm's optimization of data association also improves the stability of multi-target tracking. Specifically, the figure shows that the ID of the target with ID 3 remains unchanged even when occluded. This phenomenon indicates that the optimized Kalman filter and data association strategy enhance the algorithm's target association capability, especially in complex scenes. It can effectively handle the similarity between targets and the influence of occlusion, thereby avoiding the ID jump problem caused by occlusion or other complex situations and ensuring the consistency of multi-target tracking.

[0071] Finally, in terms of overall results, almost no pedestrians were missed in the experiment, further demonstrating the superiority of the algorithm of this invention in terms of multi-target tracking accuracy.

[0072] like Figure 7 The figures show the target tracking results of the algorithm on the MOT20-02 video sequence. Figures (a) and (b) show that the algorithm can still accurately detect the target with ID 27 even when it is occluded by obstacles, demonstrating its advantage in occluded scenarios. Figures (b) and (c) show that targets with IDs 16, 6, and 24 did not exhibit ID jump issues when walking in dense crowds, indicating that the improved algorithm also has advantages in target association. Compared to the MOT17-04 sequence, the targets in the MOT20-02 video sequence are denser, and target occlusion and overlap occur more frequently. Despite this, the algorithm proposed in this invention still maintains high accuracy in this video sequence with very few missed detections. This shows that the algorithm not only achieves excellent results on the MOT17 dataset but also demonstrates strong accuracy and robustness on the more challenging MOT20 dataset.

[0073] Multi-target tracking test and verification of RK3588 platform The improved ByteTrack-RK algorithm was deployed on the RK3588 platform for testing and verification. Target tracking experiments were conducted on 720P and 1080P video streams to verify whether the platform meets real-time requirements for multi-target tracking tasks in practical applications. The RK3588 platform's automatic frequency tuning function was not enabled in this experiment; a fixed frequency was used instead. The accuracy and inference speed of the target tracking algorithm are shown in Table 5. The table shows that the improved ByteTrack-RK algorithm proposed in this invention outperforms the benchmark algorithm in both accuracy and inference speed.

[0074] Table 5 Target Tracking Algorithm Results like Figure 8The experimental results of the algorithm for target tracking in aerial video streams on the RK3588 platform are shown. As can be seen from Figure (b), the ByteTrack-RK algorithm achieves an inference speed of 45 frames per second (fps) in 720P video streams, while in Figure (d), the inference speed reaches 35 fps in 1080P video streams. Compared to the original, unmodified ByteTrack algorithm, the inference speed is significantly improved. The ByteTrack-RK algorithm improves processing speed by optimizing the detector and improving the matching strategy, reducing the computational burden while maintaining accuracy. This allows the algorithm to maintain a high frame rate when processing high-definition video streams, adapting to more complex scenes and higher-resolution video streams. Furthermore, the experiments show that the algorithm can achieve good real-time performance without sacrificing accuracy at different video stream resolutions, providing a more practical solution for applications in embedded systems and other environments with limited computing resources. These results fully demonstrate the superior performance of the ByteTrack-RK algorithm in multi-target tracking tasks, especially its strong stability and robustness in complex video stream scenarios.

Claims

1. A multi-target tracking method, characterized in that, include: Obtain multiple detection boxes and their confidence scores output by the target detector for the current video frame; Based on the distribution characteristics of the confidence level, the confidence threshold of the current frame is dynamically determined, and the detection box is divided into a high-confidence set and a low-confidence set according to the threshold, so as to perform adaptive quality screening of the detection results. An adaptive Kalman filter is used to predict the state of an existing target trajectory. The process noise covariance (Q) and observation noise covariance (R) of the adaptive Kalman filter are estimated and adjusted online based on the prediction information and observation residuals. Based on the distance intersection-union ratio (DIoU) metric, the detection boxes in the high-confidence set are matched and associated with the trajectories predicted by the adaptive Kalman filter for the first time. Trajectories that were not associated after the first match are matched and associated with detection boxes in the low-confidence set for the second time to recover potential targets that may have been lost due to the adaptive quality screening. Output the target tracking results for the current frame.

2. A multi-target tracking system, characterized in that, include: The detection result acquisition module is used to acquire multiple detection boxes and their confidence scores output by the target detector for the current video frame; An adaptive threshold module is used to dynamically determine the confidence threshold of the current frame based on the distribution characteristics of the confidence level, and divide the detection box into a high-confidence set and a low-confidence set according to the threshold. An adaptive Kalman filter prediction module is used to predict the state of an existing target trajectory using an adaptive Kalman filter, wherein the process noise covariance (Q) and observation noise covariance (R) of the adaptive Kalman filter are estimated and adjusted online based on the prediction information and observation residuals; The data association module is used to: perform a first matching association between the detection boxes in the high-confidence set and the predicted trajectories based on the distance intersection-union ratio (DIoU) metric; and perform a second matching association between the trajectories that were not associated after the first matching and the detection boxes in the low-confidence set. The results output module is used to output the target tracking results for the current frame.

3. The method or system according to claim 1 or 2, characterized in that, The target detector is a lightweight neural network model that reduces computation by replacing some standard convolutional layers with Ghost modules and includes a shallow detection head for small target detection, with an overall parameter count not exceeding 12.2M.

4. The method or system according to claim 3, characterized in that, The resolution of the shallow detection head is 160×160.

5. The method or system according to claim 1 or 2, characterized in that, The dynamic determination of the adaptive confidence threshold includes: arranging the confidence scores of all detection boxes in a single frame in descending order to form a sequence; calculating the forward difference of the sequence; and determining the confidence score corresponding to the maximum forward difference as the dynamic threshold of the current frame.

6. The method or system according to claim 1 or 2, characterized in that, The smoothing coefficient α of the low-pass filter is 0.

3.

7. The method according to claim 1 or the system according to claim 2, characterized in that, The formula for calculating the Distance Intersection over Union (DIoU) is as follows: ,in, and These are the center points of the predicted bounding box and the ground truth bounding box, respectively. For Euclidean distance, It is the diagonal length of the smallest bounding rectangle containing the two frames.

8. The method or system according to claim 1 or 2, characterized in that, When executed on the RK3588 platform, the method or system achieves a target tracking speed of no less than 45 FPS for a 720P resolution video stream.

9. The system according to claim 2, characterized in that, The multi-core NPU asynchronous parallel inference adopts a pipeline strategy based on functional partitioning, wherein: the first NPU core is dedicated to performing forward inference of the target detector; the second NPU core performs data association calculation and Kalman filtering state updates in parallel; and the third NPU core is responsible for caching and managing the state queue of historical trajectories.

Images