Method suitable for evaluating tracking performance of object shadow target

Abstract

The invention discloses a shadow target tracking performance evaluation method suitable for multiple scenes, and belongs to the technical field of artificial intelligence and computer vision. According to the method, a TSD reference data set containing 200 video sequences and about more than 50000 frames is constructed aiming at the defect of shadow tracking of an existing evaluation system under multiple scenes, bounding box labeling is carried out on each frame, and the frames are divided into four shadow scene types including planes, snowfields, grassland and mountainous regions. The evaluation framework adopts a mixed protocol, and tests 20 mainstream tracking algorithms in combination with the precision rate and the success rate of single-pass evaluation and an average precision index in four scenes. Experimental results show that few algorithms are excellent in performance, but all the algorithms are not prominent in performance, and the performance bottleneck of an existing algorithm is revealed. According to the method, the shadow data set under multiple scenes is constructed, a standardized evaluation scheme is provided for optimization of an object shadow target tracking algorithm, and application of related technologies in unmanned driving technologies and other scenes can be promoted.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer vision technology, and particularly relates to a target tracking performance evaluation method for complex scenes such as unmanned aerial vehicles and automatic driving. Such application environments often have challenges such as complex dynamic changes and reflection interference, and put forward very high requirements for the robustness of tracking algorithms. Unlike conventional target tracking, shadow tracking needs to cope with special problems such as low-contrast features, target dynamic deformation, and environmental light interference.

[0002] The core difficulty of shadow tracking lies in the nonlinear changes of shape, scale and direction of the shadow during the movement of the object, which has inherent similarity with the class switching and appearance mutation problems handled in general target tracking. In order to scientifically evaluate the robustness and generalization performance of the target tracking algorithm under such complex conditions, the present application aims to construct a unified evaluation mechanism integrating data collection, labeling, enhancement, evaluation and feedback for factors such as feature mutation caused by shadow reflection deformation under different scenes. BACKGROUND

[0003] In recent years, with the rapid development of automatic driving, robot navigation, and virtual reality, shadow tracking technology has become increasingly important. MIT's ShadowCam system can detect moving vehicles or pedestrians at corners up to 0.5-1 seconds in advance by analyzing the subtle changes in ground shadows, responding faster than traditional LiDAR. Through signal amplification technology, the system can process shadows in low light conditions, maintaining high accuracy in simulated night environments and avoiding collisions caused by visual obstructions. Adobe and Jia's team proposed the GenProp framework, which can track objects and their shadows, reflections, and other "side effects" to achieve realistic video editing. The system supports dynamic rendering of special effects such as fire and smoke, and ensures that shadows change in real time with the light source, solving the problem of insufficient physical consistency in traditional editing. Shadow tracking technology opens up new paths in safety, creativity, and automation through reverse thinking. With the integration of low-light enhancement, physical modeling, and real-time computing technology, its application boundaries will further expand, especially in automatic driving prediction and all-time unmanned aerial vehicle operations, becoming a key tool for breaking through the perception bottleneck. Shadow morphology is highly nonlinear due to changes in lighting conditions. Second, the texture interference of the projection surface causes the target feature expression to degrade significantly. Third, long-term occlusion causes apparent mutations, significantly increasing tracking difficulty. These characteristics require the tracking algorithm to have: 1) strong adaptability to non-rigid deformation, 2) feature extraction mechanism against surface interference, and 3) reliable long-term memory modeling architecture. This work can stimulate researchers' interest in shadow tracking in key scenarios such as intelligent monitoring, autonomous driving, and augmented reality, and further promote the development of visual tracking technology to a higher level of perception understanding. Shadows in the four different ground scenarios proposed will exhibit significantly different characteristics, influenced by factors such as ground material, lighting conditions, and environmental reflections. In different ground environments, shadows exhibit distinct characteristics: 1) Snow ground shadows appear with a blue tint due to the influence of scattered light from the blue sky, with sharp boundaries. When objects move quickly, the shadows produced by the snowflakes carried by the fast-moving objects can cause blurring, affecting the tracking results. 2) Grassland shadows have a green tint due to vegetation reflection, with soft boundaries. The depth of the shadow is directly affected by the density of the vegetation. When shadows appear on the grassland, the wind can cause the grass to sway, causing the shadow to change and move in real time, affecting the tracking results. 3) Mountain shadows appear in dark gray to black tones and irregular shapes due to the complex terrain. The rugged terrain can cause complex changes in object shadows, leading to a significant degradation of target feature expression. 4) Desert shadows appear in yellow-brown tones due to sand reflection, with clear boundaries. Compared to the previous three scenarios, the complexity of shape change is slightly reduced. These characteristic differences make the visual performance and practical applications of shadows in different scenarios unique and worthy of research.

[0004] Visual object tracking, as a core research direction of computer vision, aims to continuously locate and identify specific objects in video sequences, and has wide applications in key fields such as intelligent monitoring, autonomous driving, augmented reality, and human-computer interaction. In recent years, with the development of deep neural networks and large-scale video datasets, visual tracking technology has made significant progress. However, existing researches mostly focus on tracking rigid and non-deformable objects, and there is still a lack of systematic attention to the tracking problem of shadows, a special visual element.

[0005] Currently widely used large tracking datasets such as GOT-10k, LaSOT, OTB, VOT, and TrackingNet provide important support for the development and evaluation of general object tracking algorithms. These datasets cover diverse target categories, motion patterns, and scene types, and establish standardized evaluation systems, but their collection conditions are mostly based on conventional scenes with complete and visible targets, and do not include shadow tracking-related content. This leads to performance degradation of models trained based on such datasets when facing shadow targets in real complex scenes.

[0006] From the method architecture, mainstream trackers can be roughly divided into three categories: first, CNN-based models such as SiamAPN++, SiamCAR, SiamGAT, which rely on convolutional neural networks to extract spatial features and perform similarity matching, have good real-time performance, but their robustness in complex visual conditions is still limited; second, CNN-Transformer hybrid models such as DCPT, STARK, UncTrack, which combine the local feature extraction capability of CNN with the global context modeling advantage of Transformer, attempt to balance between local details and global relationships; third, Transformer-based trackers such as ARTrack, AVTrack, AQATrack, DropTrack, HIPTrack, ODTrack, which are emerging categories with more advanced structures, show certain potential in handling target occlusion or disappearance and deformation. However, the performance differences and internal bottlenecks of the above methods in scenes with severe target deformation or significant shadow interference have not been systematically revealed, and there is a lack of targeted evaluation framework and optimization guidance.

[0007] In summary, although existing datasets and algorithms have promoted the development of the object tracking field to some extent, there is still a significant gap in shadow tracking. Existing benchmarks do not cover shadow-related scenes, and the performance of mainstream trackers in this joint task lacks systematic evaluation, leading to a lack of unified and effective evaluation criteria for shadow object tracking algorithm development, which restricts its application in real complex environments. Therefore, it is of great research significance and application value to construct a shadow tracking benchmark dataset suitable for multiple environmental conditions and a supporting evaluation system. SUMMARY

[0008] The present application aims to provide a shadow target tracking (TSD) benchmark construction and evaluation method, which realizes comprehensive evaluation of shadow tracking of tracking algorithms by constructing fine annotation dataset of real scenes and combining evaluation framework, and provides quantitative basis for algorithm optimization.

[0009] (1) Dataset construction

[0010] 200 real scene video sequences are collected, containing more than 50000 frames, including four types of scenes: flat ground, snow ground, grassland and mountainous area, and the video sources include handheld device shooting and camera recording, etc., to ensure the authenticity and application relevance of the data;

[0011] Each frame is annotated:

[0012] According to the deformation scene of the shadow, it is divided into four categories: planar deformation (LG), snow deformation (SN), grassland deformation (LEA) and mountainous deformation (MT).

[0013] (2) Evaluation framework design

[0014] A hybrid evaluation protocol is adopted, combining one-pass evaluation (OPE) and average precision (AP) indicators:

[0015] The OPE protocol includes precision rate (PRC) and success rate (SUC), which respectively measure the target center positioning error and the predicted box intersection over union (IoU) performance;

[0016] The AP indicator calculates the shadow under four types of scenes, quantifying the shadow tracking ability of the algorithm under different scenes;

[0017] Benchmark test is conducted on 20 mainstream tracking algorithms, covering CNN-based trackers (such as SiamCAR, etc.), CNN-Transformer hybrid trackers (such as UncTrack, DCPT), Transformer-based trackers (such as HIPTrack, ProContext), etc., to analyze the performance bottleneck of different architectures under different shadow deformation scenes.

[0018] (3) Key innovations

[0019] The benchmark dataset of shadows under different scenes is constructed for the first time, filling the gap in existing technologies and supporting robustness evaluation of algorithms under complex visual conditions;

[0020] The four types of shadow deformation scene classification standards and hierarchical evaluation method are proposed, realizing fine analysis of the non-modal reasoning ability of the algorithm;

[0021] A multi-dimensional evaluation system is established, and OPE and AP indicators are combined to comprehensively reflect the algorithm performance, so as to provide a clear direction for tracking model optimization.

[0022] Through the method, the performance of the tracking algorithm in different shadow deformation scenes can be quantitatively evaluated, the optimization of the algorithm in the feature extraction and structure inference direction is guided, and the deployment effect of the model in the actual scene such as automatic driving technology is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] FIG. 1 is an example of shadows of the TSD dataset under different scenes

[0024] FIG. 2 is a comparison of OPE performance of 20 tracking algorithms on the entire evaluation benchmark

[0025] FIG. 3 is a comparison of AP performance of the top 7 tracking algorithms with higher performance under four different shadow deformation conditions DETAILED DESCRIPTION

[0026] The shadow target tracking benchmark construction and evaluation method of the present application comprises the following steps:

[0027] (1) Data set construction process data acquisition and screening: real low light scene videos are collected from Bilibili, YouTube, TikTok and other platforms, covering night streets, underground passages, indoor weak light environments and the like, and the video sources include handheld devices, vehicle-mounted cameras and the like, to ensure the authenticity of the scene; 200 sequences (about 50000 frames) are screened, containing shadow targets of types such as pedestrians, vehicles and animals; the labeling process is as follows: firstly, adaptive filtering is used for flat ground to eliminate jaggies; normal perturbation and light transmission simulation are introduced for grassland to achieve soft penumbra effect; for highly reflective snow-covered land, environmental light shielding and indirect light estimation are combined to suppress dead black and compensate for cold tones; for rugged mountainous areas, the depth offset is dynamically adjusted according to the terrain normal, and the shadow is corrected by combining the height map, and the above algorithms can automatically solve specific shadow defects according to the scene characteristics, improve the rendering realism, and optimize the calculation efficiency, and then a professional team is used for labeling to ensure that the labeling accuracy is ≥95%.

[0028] (2) Evaluation indicators and mathematical formula precision rate (PRC): measure the target center positioning accuracy, the calculation formula is:

[0029]

[0030] wherein N is the total number of frames, and are the predicted center and the true center coordinates of the i-th frame, is the distance threshold (usually set to 20 pixels), is an indicator function (1 if the condition is met, otherwise 0); success rate (SUC): measures the overlap between the predicted box and the true box, and the calculation formula is:

[0031]

[0032] wherein, and are the predicted box and the true box of the ith frame, respectively, is the IoU threshold (usually set to 0.5), and IoU is the intersection over union (the ratio of the intersection area to the union area of the two boxes); average precision (AP): calculated by layering according to the occlusion level, and the formula is:

[0033]

[0034] wherein, r represents the level of shadow deformation in different environmental scenes, and are the precision and recall at the kth threshold, respectively, K is the total number of thresholds, and AP is the area under the precision-recall curve.

[0035] (3) Algorithm evaluation and result analysis Experimental setup: 20 tracking algorithms were run on a unified hardware platform using the default open source implementation without task-specific fine-tuning to ensure fairness of evaluation;

[0036] In performance tests, ProContext, SeqTrack, and AVTrack performed best in overall OPE indicators, with success rates of 56.7%, 55.8%, and 54.6%, respectively, and precision rates of 63.9%, 65.7%, and 60.8%, respectively. AP hierarchical evaluation of four types of scenes (snow SN, mountain MT, grass LEA, and flat LG) showed that algorithm performance was deeply coupled with scene physical characteristics: snow was the most challenging due to strong light reflection and low texture contrast (average success rate of only 45.2%), while SeqTrack led with a success rate of 65.1% and a precision rate of 68.5% due to its time sequence modeling advantage. In mountainous scenes, AVTrack led with a success rate of 51.4%, but its precision rate (54.3%) had a shortcoming. In grassland scenes, texture modeling was highlighted, with DCPT leading with a success rate of 56.9%. In flat ground scenes, ProContext led with a success rate of 60.0%, and SeqTrack set a record with a precision rate of 71.1%. Performance differentiation exposed inherent algorithm defects, such as TransT's significant precision-success rate gap (4.7%) in mountainous scenes and SiamGAT's poor performance in all four types of scenes. This proved that insufficient model capacity cannot cope with environmental complexity. These findings led to the need for scene customization: 1) snow needs to integrate anti-reflective modeling (such as SeqTrack's time sequence correlation learning) and multispectral information; 2) mountains should strengthen motion trajectory prediction (ProContext's core) and integrate inertial measurement unit compensation for sudden terrain changes; 3) grasslands must integrate dynamic texture response modules (DCPT mechanism) to suppress vegetation disturbance.

[0037] In conclusion, future breakthroughs require three-dimensional linkage: integrating sequence modeling (SeqTrack), cross-modal perception (AVTrack), and context prediction (ProContext) in architecture; embedding scene physical priors (such as snow reflectance model / mountain motion dynamics); and developing attention gate mechanisms to address Transformer deficiencies to achieve robust tracking in complex environments.

Claims

1. A method for evaluating the performance of object shadow target tracking, characterized in that, Specifically, the following steps are included: S1: Collect video data containing people, animals, vehicles, etc. The data source includes real scene videos reflecting shadows in environments such as flat ground and rugged ground; S2: Add bounding boxes to the shadow video frames. The annotator generates an accurate axis-aligned bounding box for each frame based on the initial target position and the position of the shadow, which fully includes all visible parts of the target. When the target is not visible or is completely occluded, it is labeled as "out of view" or "completely occluded". S3: Based on the dynamic deformation ratio of the target's shadow in each frame, it is divided into four shadow scene types: Plane Deformation (LG), Snow Terrain Deformation (SN), Grassland Deformation (LEA), and Mountain Terrain Deformation (MT). S4: Construct an evaluation dataset and introduce an evaluation index system to perform performance tests on various target tracking algorithms. The indexes include accuracy (PRC) and success rate (SUC) under One Pass Evaluation (OPE), as well as average accuracy (AP) under four different scene types. S5: Outputs a multi-dimensional evaluation report, recording the performance of each algorithm under different environmental conditions and reflections, for research on visual tracking algorithms for shadow targets in multiple scenarios.

2. The method according to claim 1, characterized in that, When annotating bounding boxes, the specific location of the target in the shadow area of ​​the current frame is considered. The annotation is done manually by annotation experts with target recognition experience, and the bounding boxes are drawn simultaneously.

3. The method according to claim 1, characterized in that, The evaluation process in S4 and S5 includes quantitatively evaluating the tracking algorithm performance using the One Pass Evaluation (OPE) metric and Average Precision (AP); wherein the OPE metric includes: (1) Precision (PRC): Statistically determine whether the Euclidean distance between the predicted target center point and the actual center point in each frame is less than a set threshold, preferably 20 pixels; (2) Success rate (SUC): Whether the intersection-union ratio (IoU) between the predicted bounding box and the actual bounding box is greater than 0.5; The AP metric is calculated separately for shadow deformation under four different scene types: planar deformation (LG), snow terrain deformation (SN), grassland deformation (LEA), and mountain terrain deformation (MT) to evaluate the robustness of the algorithm under different environmental conditions. The final evaluation report includes AP analysis tables for deformation groups under different environments, ranking charts of each algorithm, examples of mislocalization images, and improvement suggestions for performance bottlenecks to support subsequent algorithm optimization and comparative studies.

Images